Oct 30 16:19:26.246930 ip-10-0-137-219 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:19:26.653102 ip-10-0-137-219 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:19:26.653102 ip-10-0-137-219 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:19:26.653102 ip-10-0-137-219 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:19:26.653102 ip-10-0-137-219 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:19:26.659563 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659477 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:26.659563 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659564 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659568 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659572 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659575 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659578 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659581 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659584 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659587 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659590 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659592 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659596 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659598 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659601 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659604 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659606 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659609 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659612 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659614 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659617 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659619 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:26.659638 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659622 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659624 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659627 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659630 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659633 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659635 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659645 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659648 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659651 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659672 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659675 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659678 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659680 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659683 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659686 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659688 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659691 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659693 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659696 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659698 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:26.660108 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659701 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659703 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659706 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659708 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659711 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659713 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659716 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659718 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659721 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659725 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659728 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659731 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659734 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659736 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659742 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659744 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659747 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659749 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659752 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659755 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:26.660584 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659757 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659760 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659763 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659765 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659769 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659774 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659777 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659780 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659783 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659786 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659789 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659792 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659794 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659797 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659799 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659802 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659804 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659807 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:26.661069 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659809 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659814 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659818 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659820 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659823 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659826 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659829 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659832 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659836 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659839 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659841 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659844 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659852 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659856 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659859 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659861 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659864 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659866 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659869 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:26.661494 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659871 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:26.661981 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659874 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:26.661981 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659876 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:26.661981 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659879 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:26.661981 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659882 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:26.661981 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659884 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:26.661981 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659887 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:26.661981 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.659889 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:26.662316 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662302 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:26.662316 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662315 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662319 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662322 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662324 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662327 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662330 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662332 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662335 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662337 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662340 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662343 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662346 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662349 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662351 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662354 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662356 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662359 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662362 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662364 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:26.662368 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662367 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662369 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662372 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662375 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662377 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662380 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662383 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662386 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662388 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662391 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662393 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662396 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662399 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662402 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662406 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662409 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662412 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662415 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662418 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662421 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:26.662846 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662424 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662426 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662429 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662431 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662442 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662445 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662447 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662451 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662455 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662457 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662460 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662462 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662465 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662467 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662475 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662478 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662481 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662483 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662486 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:26.663320 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662489 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662491 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662494 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662496 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662499 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662501 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662504 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662507 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662509 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662512 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662515 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662517 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662520 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662522 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662525 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662527 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662530 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662533 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662536 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:26.663811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662538 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662541 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662543 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662547 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662550 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662552 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662555 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662557 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662563 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662566 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662568 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662571 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662574 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662577 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662579 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662582 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662584 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662587 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662589 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662592 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:26.664269 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662594 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662597 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662599 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662602 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662605 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662608 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662610 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.662613 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663110 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663115 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663118 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663122 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663124 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663127 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663130 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663132 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663135 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663138 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663141 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:26.664787 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663143 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663149 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663152 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663155 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663157 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663160 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663162 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663165 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663168 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663171 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663175 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663178 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663181 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663184 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663186 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663189 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663191 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663193 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663196 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:26.665233 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663199 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663201 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663204 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663206 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663209 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663211 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663214 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663217 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663219 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663222 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663225 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663227 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663230 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663233 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663237 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663240 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663242 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663245 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663247 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:26.665690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663250 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663253 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663255 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663258 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663260 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663263 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663265 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663268 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663271 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663273 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663276 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663278 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663281 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663283 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663286 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663288 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663291 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663294 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663297 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663300 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:26.666155 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663303 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663305 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663309 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663314 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663317 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663320 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663323 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663327 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663330 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663333 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663336 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663338 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663341 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663344 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663346 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663349 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663352 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663355 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663358 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:26.666624 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663360 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663363 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663365 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663368 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663370 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663373 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663375 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663378 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663380 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663383 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663385 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663388 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663390 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663393 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663395 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663398 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663401 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.663403 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664252 2583 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664262 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664272 2583 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:19:26.667090 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664277 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664282 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664285 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664290 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664301 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664305 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664308 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664311 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664315 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664318 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664322 2583 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664325 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664328 2583 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664330 2583 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664333 2583 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664339 2583 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664342 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664345 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664348 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664352 2583 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664356 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664359 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664362 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:19:26.667629 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664365 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664368 2583 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664372 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664375 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664378 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664381 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664386 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664389 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664393 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664396 2583 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664399 2583 flags.go:64] FLAG: --enable-server="true" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664402 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664407 2583 flags.go:64] FLAG: --event-burst="100" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664410 2583 flags.go:64] FLAG: --event-qps="50" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664413 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664416 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664419 2583 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664424 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664427 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664431 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664434 2583 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664437 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664440 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664443 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664446 2583 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:19:26.668179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664449 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664452 2583 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664455 2583 flags.go:64] FLAG: --feature-gates="" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664459 2583 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664463 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664466 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664469 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664472 2583 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664475 2583 flags.go:64] FLAG: --help="false" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664478 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-137-219.ec2.internal" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664482 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664485 2583 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664488 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664492 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664495 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664498 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664502 2583 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664505 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664507 2583 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664511 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664514 2583 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664517 2583 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664520 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664522 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:19:26.668857 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664526 2583 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664529 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664531 2583 flags.go:64] FLAG: --lock-file="" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664534 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664537 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664540 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664546 2583 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664548 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664551 2583 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664554 2583 flags.go:64] FLAG: --logging-format="text" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664558 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664561 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664564 2583 flags.go:64] FLAG: --manifest-url="" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664567 2583 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664571 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664575 2583 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664579 2583 flags.go:64] FLAG: --max-pods="110" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664581 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664584 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664587 2583 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664591 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664594 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664597 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664600 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664608 2583 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:19:26.669432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664611 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664614 2583 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664617 2583 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664620 2583 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664622 2583 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664626 2583 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664628 2583 flags.go:64] FLAG: --port="10250" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664631 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664634 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e7ad6bbabb8f54d9" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664638 2583 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664641 2583 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664644 2583 flags.go:64] FLAG: --register-node="true" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664647 2583 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664651 2583 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664669 2583 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664672 2583 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664675 2583 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664679 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664682 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664685 2583 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664688 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664691 2583 flags.go:64] FLAG: --runonce="false" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664694 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664697 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664700 2583 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664703 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664706 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:19:26.670031 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664709 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664712 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664716 2583 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664719 2583 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664722 2583 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664725 2583 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664728 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664731 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664734 2583 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664737 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664743 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664746 2583 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664749 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664753 2583 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664756 2583 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664759 2583 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664762 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664765 2583 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664768 2583 flags.go:64] FLAG: --v="2" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664773 2583 flags.go:64] FLAG: --version="false" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664777 2583 flags.go:64] FLAG: --vmodule="" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664781 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.664785 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664936 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664940 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:26.670665 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664942 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664945 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664948 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664951 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664953 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664956 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664959 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664962 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664965 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664968 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664973 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664975 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664978 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664981 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664984 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664986 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664989 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664991 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664994 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:26.671279 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.664999 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665002 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665007 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665010 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665013 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665015 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665018 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665021 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665023 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665026 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665029 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665031 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665035 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665038 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665041 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665043 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665046 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665048 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665051 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665054 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:26.671747 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665056 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665059 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665062 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665066 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665069 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665072 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665074 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665077 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665079 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665082 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665085 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665087 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665090 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665092 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665095 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665098 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665100 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665103 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665105 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:26.672393 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665108 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665111 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665113 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665116 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665118 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665122 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665124 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665127 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665129 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665132 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665134 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665137 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665139 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665142 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665144 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665147 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665151 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665157 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665160 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665163 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:26.672923 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665166 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665170 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665174 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665177 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665180 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665182 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665185 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665188 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665190 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665193 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665195 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665198 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665200 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665203 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665205 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665208 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665210 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665214 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665217 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:26.673391 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665219 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665222 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665225 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665227 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665230 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665232 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665235 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.665261 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665297 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665302 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665307 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665316 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665319 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665322 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665325 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665329 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:26.673854 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665332 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665336 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665338 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665341 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665344 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665346 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665349 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665351 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665354 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665357 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665359 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665362 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665364 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665367 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665372 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665375 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665378 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665380 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665383 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665386 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:26.674240 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665388 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665391 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665393 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665396 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665399 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665401 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665405 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665407 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665415 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665418 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665421 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665423 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665426 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665428 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665431 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665434 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665436 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665439 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665441 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665444 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:26.674714 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665447 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665449 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665452 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665454 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665457 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665459 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665463 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665466 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665468 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665471 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665474 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665476 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665479 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665481 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665484 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665486 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665489 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665492 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665497 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:26.675179 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665502 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665505 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665509 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665511 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665514 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665516 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665519 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665522 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665524 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665526 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665529 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665532 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665534 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665537 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665539 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665542 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665545 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665547 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665550 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:26.675626 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665554 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665556 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665559 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665561 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665564 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665566 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665569 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665571 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665574 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665577 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665579 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665582 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665586 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665588 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665591 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665594 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665597 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665600 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665603 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.665605 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:26.676076 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.673063 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.673080 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.673875 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.673881 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673939 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673944 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673948 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673951 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673955 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673958 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673961 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673964 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673967 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673969 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673972 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673975 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673978 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673980 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673983 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673986 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:26.676559 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673989 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673992 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673994 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673997 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.673999 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674002 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674005 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674007 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674010 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674012 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674015 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674018 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674021 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674025 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674027 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674030 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674033 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674035 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674038 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:26.677406 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674041 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674043 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674046 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674049 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674051 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674054 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674057 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674059 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674062 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674065 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674067 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674070 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674074 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674076 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674079 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674082 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674084 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674087 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674090 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:26.678145 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674092 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674095 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674097 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674100 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674104 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674108 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674112 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674115 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674119 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674122 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674126 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674130 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674134 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674137 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674139 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674142 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674145 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674147 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674150 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:26.678648 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674152 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674155 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674157 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674160 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674162 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674165 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674168 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674171 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674174 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674177 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674179 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674182 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674184 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674187 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674189 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674192 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674194 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674197 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674199 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674202 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:26.679142 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674205 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674208 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674210 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674213 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674215 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674218 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674220 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674223 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674225 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674228 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674231 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674233 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674236 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.674264 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674429 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674434 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:26.679620 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674437 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674441 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674445 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674450 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674454 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674457 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674459 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674462 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674465 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674467 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674470 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674473 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674475 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674478 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674480 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674483 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674486 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674489 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674491 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674494 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:26.680022 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674497 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674500 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674503 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674507 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674509 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674512 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674515 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674518 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674520 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674523 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674525 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674528 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674530 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674533 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674536 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674538 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674541 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674544 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674546 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:26.680501 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674549 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674551 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674554 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674556 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674559 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674561 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674564 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674566 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674569 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674571 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674574 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674577 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674579 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674582 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674585 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674588 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674590 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674593 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674595 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674598 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:26.681056 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674600 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674603 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674606 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674608 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674611 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674613 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674616 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674632 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674636 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674640 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674643 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674645 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674648 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674650 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674664 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674668 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674670 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674673 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674675 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674678 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:26.681526 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674681 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674684 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674687 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674689 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674692 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674694 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674698 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674700 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674703 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674705 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674708 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674711 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674713 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674716 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674718 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674722 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674724 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674727 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674730 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:26.682037 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674733 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674736 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674738 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674741 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674743 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674746 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.674770 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674803 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674807 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674810 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674814 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674817 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674820 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674823 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674826 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674831 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:19:26.682505 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674833 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674836 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674839 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674841 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674845 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674847 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674850 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674852 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674856 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674860 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674864 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674866 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674869 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674872 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674874 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674877 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674880 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674883 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674885 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:19:26.682892 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674888 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674890 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674893 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674896 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674898 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674901 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674903 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674906 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674908 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674911 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674913 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674916 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674920 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674923 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674925 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674928 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674930 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674933 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674936 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674939 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:19:26.683337 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674941 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674944 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674946 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674949 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674952 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674954 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674956 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674959 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674962 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674965 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674968 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674970 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674973 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674975 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674978 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674980 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674983 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674986 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674988 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674991 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:19:26.683819 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674994 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674996 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.674999 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675001 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675005 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675009 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675011 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675014 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675016 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675019 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675022 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675025 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675027 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675030 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675033 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675035 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675038 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675040 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675043 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:19:26.684285 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675046 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675048 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675051 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675053 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675056 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675058 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675061 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675064 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675066 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675069 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675072 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675074 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675077 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675079 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675082 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675084 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675089 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675094 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:26.675097 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:19:26.684748 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.675166 2583 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:19:26.685209 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.677759 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 16:19:26.685209 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.679028 2583 server.go:1014] "Starting client certificate rotation" Oct 30 16:19:26.685209 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.679144 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:19:26.685209 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.679226 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:19:26.685209 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.680933 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:19:26.685209 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.682049 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:19:26.695081 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.695060 2583 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:19:26.698410 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.698394 2583 log.go:25] "Validated CRI v1 image API" Oct 30 16:19:26.699227 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.699201 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:19:26.703632 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.703613 2583 fs.go:135] Filesystem UUIDs: map[379128d4-6b47-4be8-a1db-fbeb01481d55:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c1556daf-498c-4e61-a4c5-127f44fbd06b:/dev/nvme0n1p3] Oct 30 16:19:26.703704 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.703631 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 16:19:26.708670 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.708560 2583 manager.go:217] Machine: {Timestamp:2026-10-30 16:19:26.707297685 +0000 UTC m=+0.340169006 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098232 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d0d282908157e75aacc4e2f029971 SystemUUID:ec2d0d28-2908-157e-75aa-cc4e2f029971 BootID:59dcdc88-0b66-496a-b3ae-76cef16fb0fd Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:56:ba:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:56:ba:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:47:9d:fc:9c:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:19:26.708749 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.708677 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:19:26.708793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.708781 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:19:26.712402 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.712376 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:19:26.712533 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.712404 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-219.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:19:26.713088 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.713078 2583 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:19:26.713122 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.713090 2583 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:19:26.713122 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.713114 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:19:26.713174 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.713127 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:19:26.714446 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.714436 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:19:26.717485 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.717474 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:19:26.717604 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.717594 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:19:26.718395 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.718384 2583 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:19:26.718432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.718400 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:19:26.718432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.718411 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:19:26.718432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.718420 2583 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:19:26.718432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.718429 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:19:26.718789 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.718777 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:19:26.721070 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.721055 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:19:26.727864 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.727847 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:19:26.728674 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.728648 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:19:26.728717 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.728692 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:19:26.728851 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.728840 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:19:26.728880 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.728856 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:19:26.728880 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.728863 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:19:26.728880 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.728869 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:19:26.728880 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.728875 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:19:26.728984 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.728881 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:19:26.729513 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.729504 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:19:26.729541 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.729514 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:19:26.729541 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.729521 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:19:26.729541 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.729527 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:19:26.729541 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.729535 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:19:26.729744 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.729545 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:19:26.730181 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.730172 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:19:26.730211 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.730183 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:19:26.733925 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.733912 2583 server.go:1263] "Started kubelet" Oct 30 16:19:26.734054 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.734016 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:19:26.734211 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.734011 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:19:26.734328 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.734234 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:19:26.734688 ip-10-0-137-219 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:19:26.735587 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.735569 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:19:26.737233 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.737217 2583 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:19:26.739984 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:26.739966 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 16:19:26.742709 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.742687 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:19:26.743228 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.743207 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:19:26.743721 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:26.743698 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-219.ec2.internal\" not found" Oct 30 16:19:26.744059 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.744040 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:19:26.744129 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.744045 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:19:26.744129 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.744077 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:19:26.744229 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.744213 2583 factory.go:55] Registering systemd factory Oct 30 16:19:26.744276 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.744236 2583 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:19:26.744276 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.744271 2583 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:19:26.744381 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.744278 2583 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:19:26.744479 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.744461 2583 factory.go:153] Registering CRI-O factory Oct 30 16:19:26.744532 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.744507 2583 factory.go:223] Registration of the crio container factory successfully Oct 30 16:19:26.744584 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.744550 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:19:26.744584 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.744569 2583 factory.go:103] Registering Raw factory Oct 30 16:19:26.744584 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.744582 2583 manager.go:1199] Started watching for new ooms in manager Oct 30 16:19:26.745570 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.745551 2583 manager.go:319] Starting recovery of all containers Oct 30 16:19:26.757202 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.757178 2583 manager.go:324] Recovery completed Oct 30 16:19:26.762808 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.762791 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:26.766265 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.766246 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:26.766340 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.766277 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:26.766340 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.766290 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:26.766820 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.766807 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:26.766909 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.766809 2583 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:19:26.766909 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.766876 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:19:26.766909 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.766907 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:19:26.767496 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.767477 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:26.767574 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.767512 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:26.767574 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.767526 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:26.768342 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.768330 2583 policy_none.go:50] "Start" Oct 30 16:19:26.768391 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.768345 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:19:26.768391 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.768354 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:19:26.769400 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:26.769377 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 16:19:26.769455 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.769410 2583 policy_none.go:44] "Start" Oct 30 16:19:26.770500 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.770484 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:19:26.774257 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.774240 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8rvrh" Oct 30 16:19:26.784034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.784017 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8rvrh" Oct 30 16:19:26.790197 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.790176 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-219.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:19:26.791174 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:26.790254 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-219.ec2.internal.18e35b080b783d92 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-219.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-219.ec2.internal,},FirstTimestamp:2026-10-30 16:19:26.733884818 +0000 UTC m=+0.366756139,LastTimestamp:2026-10-30 16:19:26.733884818 +0000 UTC m=+0.366756139,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-219.ec2.internal,}" Oct 30 16:19:26.806726 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.806686 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:19:26.810827 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.810810 2583 manager.go:341] "Starting Device Plugin manager" Oct 30 16:19:26.811371 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:26.811353 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:19:26.811428 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.811371 2583 server.go:87] "Starting device plugin registration server" Oct 30 16:19:26.811632 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.811621 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:19:26.811716 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.811634 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:19:26.811843 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.811831 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:19:26.811922 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.811903 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:19:26.811922 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.811915 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:19:26.812450 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:26.812427 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:19:26.812527 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:26.812514 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-219.ec2.internal\" not found" Oct 30 16:19:26.818076 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.818062 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:19:26.819710 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.819696 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:19:26.856802 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.844668 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:19:26.888613 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.888581 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:19:26.889772 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.889753 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:19:26.889844 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.889775 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:19:26.889844 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.889798 2583 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:19:26.889844 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:26.889837 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 16:19:26.904328 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.904248 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:19:26.912343 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.912330 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:19:26.913202 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.913183 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:19:26.913288 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.913212 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:19:26.913288 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.913223 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:19:26.913288 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.913245 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-219.ec2.internal" Oct 30 16:19:26.939574 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.939551 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-219.ec2.internal" Oct 30 16:19:26.990638 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.990605 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal"] Oct 30 16:19:26.992197 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.992181 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:19:26.993072 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:26.993053 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:19:27.018594 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.018575 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:19:27.022649 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.022629 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:19:27.044236 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.044218 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:19:27.044329 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.044278 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:19:27.045441 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.045382 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/823ffa96cf0d2439ccc4c6edac7c16b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"823ffa96cf0d2439ccc4c6edac7c16b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:19:27.045507 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.045449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/823ffa96cf0d2439ccc4c6edac7c16b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"823ffa96cf0d2439ccc4c6edac7c16b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:19:27.045507 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.045477 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8e5ad7f7f3345c1093a31c3b5fc9d86-config\") pod \"kube-apiserver-proxy-ip-10-0-137-219.ec2.internal\" (UID: \"a8e5ad7f7f3345c1093a31c3b5fc9d86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:19:27.146450 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.146420 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/823ffa96cf0d2439ccc4c6edac7c16b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"823ffa96cf0d2439ccc4c6edac7c16b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:19:27.146450 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.146453 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/823ffa96cf0d2439ccc4c6edac7c16b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"823ffa96cf0d2439ccc4c6edac7c16b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:19:27.146649 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.146485 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8e5ad7f7f3345c1093a31c3b5fc9d86-config\") pod \"kube-apiserver-proxy-ip-10-0-137-219.ec2.internal\" (UID: \"a8e5ad7f7f3345c1093a31c3b5fc9d86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:19:27.146649 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.146534 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/823ffa96cf0d2439ccc4c6edac7c16b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"823ffa96cf0d2439ccc4c6edac7c16b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:19:27.146649 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.146541 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8e5ad7f7f3345c1093a31c3b5fc9d86-config\") pod \"kube-apiserver-proxy-ip-10-0-137-219.ec2.internal\" (UID: \"a8e5ad7f7f3345c1093a31c3b5fc9d86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:19:27.146649 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:27.146534 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/823ffa96cf0d2439ccc4c6edac7c16b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"823ffa96cf0d2439ccc4c6edac7c16b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:19:28.423868 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.423701 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:19:28.424794 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.424768 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:19:28.753840 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.753807 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 16:19:28.754395 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.753990 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 16:19:28.754395 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.754017 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 16:19:28.754395 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.754033 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 16:19:28.793937 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.793906 2583 apiserver.go:52] "Watching apiserver" Oct 30 16:19:28.818317 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.818293 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 16:19:28.843345 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.843321 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:19:28.843628 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.843612 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-46t9k","openshift-dns/node-resolver-7rnq2","openshift-multus/multus-qdzgt","openshift-network-diagnostics/network-check-target-mv4lc","openshift-network-operator/iptables-alerter-c2tkd","kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv","openshift-ovn-kubernetes/ovnkube-node-v4lzf","openshift-cluster-node-tuning-operator/tuned-4g45x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal","openshift-multus/multus-additional-cni-plugins-vcfvx","openshift-multus/network-metrics-daemon-4856h","kube-system/global-pull-secret-syncer-lt2pz","kube-system/konnectivity-agent-h8rzc"] Oct 30 16:19:28.845319 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.845301 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:19:28.846554 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.846535 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:19:28.847674 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.847658 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.849025 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.849006 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:28.849114 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:28.849080 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:28.850148 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.850132 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:19:28.851310 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.851294 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:28.852620 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.852606 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.854057 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.854038 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.855370 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.855354 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:28.856855 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.856841 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:28.856929 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:28.856890 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:28.858302 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.858286 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:28.858376 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:28.858350 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:28.859499 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.859484 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:19:28.863487 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.863458 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:14:27 +0000 UTC" deadline="2028-05-29 16:22:47.176239869 +0000 UTC" Oct 30 16:19:28.863487 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.863487 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13848h3m18.312757136s" Oct 30 16:19:28.863819 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.863799 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:19:28.863901 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.863825 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:19:28.863901 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.863832 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:19:28.863901 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.863840 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2smlp\"" Oct 30 16:19:28.863901 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.863835 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:19:28.864087 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.864074 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:19:28.864174 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.864161 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v666l\"" Oct 30 16:19:28.888994 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.888958 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:19:28.889105 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.889049 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6tbhv" Oct 30 16:19:28.920552 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.920527 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:19:28.924576 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.924549 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovnkube-script-lib\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.924695 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.924585 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:28.924695 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.924606 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-k8s-cni-cncf-io\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.924695 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.924624 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-run-netns\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.924695 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.924669 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-ovn\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.924873 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.924726 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rbfc\" (UniqueName: \"kubernetes.io/projected/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-kube-api-access-2rbfc\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.924873 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.924797 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30634e3d-5f01-4b9b-88ad-03893a2b444e-konnectivity-ca\") pod \"konnectivity-agent-h8rzc\" (UID: \"30634e3d-5f01-4b9b-88ad-03893a2b444e\") " pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:19:28.924945 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.924869 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysconfig\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.924945 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.924912 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysctl-conf\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.924945 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.924937 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-socket-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:28.925099 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.924991 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-registration-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:28.925099 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925022 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:28.925099 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925045 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99d2h\" (UniqueName: \"kubernetes.io/projected/fb2c72e6-372c-41c1-ba29-d47c0b521f34-kube-api-access-99d2h\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:19:28.925243 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925093 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-cni-multus\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.925243 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925132 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skbh\" (UniqueName: \"kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh\") pod \"network-check-target-mv4lc\" (UID: \"d90a91c8-f7a6-4526-a13c-ce484713b6b5\") " pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:28.925243 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925158 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-sys\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.925243 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925202 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-host\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.925549 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925276 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:28.925549 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925329 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd2b0c8e-8194-456d-8299-c43b1688404a-hosts-file\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:19:28.925549 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925355 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd2b0c8e-8194-456d-8299-c43b1688404a-tmp-dir\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:19:28.925549 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925370 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-cni-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.925549 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925394 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-cnibin\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.925549 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925408 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-cni-bin\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.925549 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925427 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6c2l\" (UniqueName: \"kubernetes.io/projected/ddebdc11-43b3-4787-a3ff-bf95150f8d27-kube-api-access-s6c2l\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:28.925549 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-modprobe-d\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.925549 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925487 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2196112-4441-49a3-b48f-1b80bf426e29-tmp\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.925549 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925501 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-kubelet\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.925549 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925521 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-log-socket\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.926018 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925558 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.926018 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925611 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-env-overrides\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.926018 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925647 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:28.926018 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925699 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblkc\" (UniqueName: \"kubernetes.io/projected/cd2b0c8e-8194-456d-8299-c43b1688404a-kube-api-access-cblkc\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:19:28.926018 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925734 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-multus-certs\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.926018 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925769 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e07b376-b6a8-4d05-b6f4-705695c0e92b-iptables-alerter-script\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:19:28.926018 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925811 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jd74\" (UniqueName: \"kubernetes.io/projected/3e07b376-b6a8-4d05-b6f4-705695c0e92b-kube-api-access-2jd74\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:19:28.926018 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925842 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovn-node-metrics-cert\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.926018 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925879 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb2c72e6-372c-41c1-ba29-d47c0b521f34-serviceca\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:19:28.926018 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925903 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e07b376-b6a8-4d05-b6f4-705695c0e92b-host-slash\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:19:28.926018 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.925923 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkz5f\" (UniqueName: \"kubernetes.io/projected/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-kube-api-access-qkz5f\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:28.926018 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926014 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-var-lib-kubelet\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926052 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-device-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926078 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-slash\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovnkube-config\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926145 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926170 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b36b8f12-981e-46a8-b422-632babc5d4a7-dbus\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926193 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-systemd\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-cnibin\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926284 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk6g4\" (UniqueName: \"kubernetes.io/projected/81a6d8ed-431a-4b9b-bd73-d13b01208249-kube-api-access-bk6g4\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926310 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b36b8f12-981e-46a8-b422-632babc5d4a7-kubelet-config\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30634e3d-5f01-4b9b-88ad-03893a2b444e-agent-certs\") pod \"konnectivity-agent-h8rzc\" (UID: \"30634e3d-5f01-4b9b-88ad-03893a2b444e\") " pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926381 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb2c72e6-372c-41c1-ba29-d47c0b521f34-host\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926398 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-socket-dir-parent\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926413 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysctl-d\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926428 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-var-lib-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.926621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926443 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-sys-fs\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926477 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-systemd-units\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926491 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-node-log\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926507 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-cni-bin\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926522 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926542 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-etc-kubernetes\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926560 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57d5q\" (UniqueName: \"kubernetes.io/projected/67036dae-0e4a-473b-af97-e0ee20d824b8-kube-api-access-57d5q\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926575 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-run\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926589 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a2196112-4441-49a3-b48f-1b80bf426e29-etc-tuned\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926604 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926620 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-system-cni-dir\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926687 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-system-cni-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926702 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-netns\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-hostroot\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926729 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-conf-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926744 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-kubernetes\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926763 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-lib-modules\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.927169 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926778 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:28.927672 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926792 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67036dae-0e4a-473b-af97-e0ee20d824b8-cni-binary-copy\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.927672 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926806 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-etc-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.927672 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926821 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.927672 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926836 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-cni-netd\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.927672 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926849 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-kubelet\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.927672 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-daemon-config\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:28.927672 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926876 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qnf9\" (UniqueName: \"kubernetes.io/projected/a2196112-4441-49a3-b48f-1b80bf426e29-kube-api-access-4qnf9\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:28.927672 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.926892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-systemd\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964418 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964480 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964519 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964536 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964599 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964613 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964642 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964649 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964681 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964714 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964782 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964814 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964844 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964929 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964933 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.964786 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:19:28.965034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.965003 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:19:28.965739 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.965068 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:19:28.967537 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.967506 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:19:28.978167 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.978148 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cpfx5\"" Oct 30 16:19:28.978167 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.978159 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8fzhw\"" Oct 30 16:19:28.978349 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.978243 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:19:28.978349 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.978264 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:19:28.978349 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.978304 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-svxjg\"" Oct 30 16:19:28.978349 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.978269 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:19:28.978349 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.978245 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8qclp\"" Oct 30 16:19:28.978349 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.978332 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6x9tv\"" Oct 30 16:19:28.978349 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.978341 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j7zxr\"" Oct 30 16:19:28.978552 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:28.978374 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dlk52\"" Oct 30 16:19:29.025905 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:29.025864 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e5ad7f7f3345c1093a31c3b5fc9d86.slice/crio-5aeb3fbd398c51d8dc64868da92878d26a596a7beb445896d953f1650f7cc0c1 WatchSource:0}: Error finding container 5aeb3fbd398c51d8dc64868da92878d26a596a7beb445896d953f1650f7cc0c1: Status 404 returned error can't find the container with id 5aeb3fbd398c51d8dc64868da92878d26a596a7beb445896d953f1650f7cc0c1 Oct 30 16:19:29.026073 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:29.026054 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823ffa96cf0d2439ccc4c6edac7c16b5.slice/crio-62e319000ca88b93d246d429f68a234857e121cbee4eb5504346e3d0ba39c0b8 WatchSource:0}: Error finding container 62e319000ca88b93d246d429f68a234857e121cbee4eb5504346e3d0ba39c0b8: Status 404 returned error can't find the container with id 62e319000ca88b93d246d429f68a234857e121cbee4eb5504346e3d0ba39c0b8 Oct 30 16:19:29.027217 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027193 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-ovn\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.027315 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027223 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rbfc\" (UniqueName: \"kubernetes.io/projected/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-kube-api-access-2rbfc\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.027315 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30634e3d-5f01-4b9b-88ad-03893a2b444e-konnectivity-ca\") pod \"konnectivity-agent-h8rzc\" (UID: \"30634e3d-5f01-4b9b-88ad-03893a2b444e\") " pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:19:29.027315 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027259 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysconfig\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.027315 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027298 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-ovn\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.027488 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027317 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysctl-conf\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.027488 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027343 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-socket-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.027488 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027365 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysconfig\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.027488 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027429 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-socket-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.027488 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027478 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysctl-conf\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.027699 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027512 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-registration-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.027699 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027537 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.027699 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027553 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99d2h\" (UniqueName: \"kubernetes.io/projected/fb2c72e6-372c-41c1-ba29-d47c0b521f34-kube-api-access-99d2h\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:19:29.027699 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027571 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-cni-multus\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.027699 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027593 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skbh\" (UniqueName: \"kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh\") pod \"network-check-target-mv4lc\" (UID: \"d90a91c8-f7a6-4526-a13c-ce484713b6b5\") " pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:29.027699 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027613 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-registration-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.027699 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-sys\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.027699 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027643 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-cni-multus\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.027699 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027657 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-host\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.027699 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027680 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:29.027699 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027699 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd2b0c8e-8194-456d-8299-c43b1688404a-hosts-file\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027706 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-sys\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd2b0c8e-8194-456d-8299-c43b1688404a-tmp-dir\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027698 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027736 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-host\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027743 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-cni-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027757 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd2b0c8e-8194-456d-8299-c43b1688404a-hosts-file\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027765 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-cnibin\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-cni-bin\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.027803 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027813 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6c2l\" (UniqueName: \"kubernetes.io/projected/ddebdc11-43b3-4787-a3ff-bf95150f8d27-kube-api-access-s6c2l\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027836 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-modprobe-d\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.027862 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs podName:ddebdc11-43b3-4787-a3ff-bf95150f8d27 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:29.52784336 +0000 UTC m=+2.085400886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs") pod "network-metrics-daemon-4856h" (UID: "ddebdc11-43b3-4787-a3ff-bf95150f8d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027868 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-cni-bin\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2196112-4441-49a3-b48f-1b80bf426e29-tmp\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027861 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-cnibin\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027912 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-kubelet\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027920 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-modprobe-d\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.028172 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027930 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-cni-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027933 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-log-socket\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.027962 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-log-socket\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028000 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028007 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-kubelet\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028030 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-env-overrides\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028046 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028057 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028082 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblkc\" (UniqueName: \"kubernetes.io/projected/cd2b0c8e-8194-456d-8299-c43b1688404a-kube-api-access-cblkc\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028108 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-multus-certs\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028131 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e07b376-b6a8-4d05-b6f4-705695c0e92b-iptables-alerter-script\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028221 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-multus-certs\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028539 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jd74\" (UniqueName: \"kubernetes.io/projected/3e07b376-b6a8-4d05-b6f4-705695c0e92b-kube-api-access-2jd74\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028560 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovn-node-metrics-cert\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028573 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb2c72e6-372c-41c1-ba29-d47c0b521f34-serviceca\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028587 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e07b376-b6a8-4d05-b6f4-705695c0e92b-host-slash\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028610 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkz5f\" (UniqueName: \"kubernetes.io/projected/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-kube-api-access-qkz5f\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.028685 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028633 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-var-lib-kubelet\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-device-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028676 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-slash\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028691 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovnkube-config\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028706 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b36b8f12-981e-46a8-b422-632babc5d4a7-dbus\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028735 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-systemd\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028740 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-var-lib-kubelet\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028748 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028766 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-cnibin\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028792 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e07b376-b6a8-4d05-b6f4-705695c0e92b-host-slash\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk6g4\" (UniqueName: \"kubernetes.io/projected/81a6d8ed-431a-4b9b-bd73-d13b01208249-kube-api-access-bk6g4\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028823 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b36b8f12-981e-46a8-b422-632babc5d4a7-kubelet-config\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028848 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30634e3d-5f01-4b9b-88ad-03893a2b444e-agent-certs\") pod \"konnectivity-agent-h8rzc\" (UID: \"30634e3d-5f01-4b9b-88ad-03893a2b444e\") " pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028870 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb2c72e6-372c-41c1-ba29-d47c0b521f34-host\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028896 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-socket-dir-parent\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028902 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30634e3d-5f01-4b9b-88ad-03893a2b444e-konnectivity-ca\") pod \"konnectivity-agent-h8rzc\" (UID: \"30634e3d-5f01-4b9b-88ad-03893a2b444e\") " pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:19:29.029411 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028919 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysctl-d\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-var-lib-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.028964 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-sys-fs\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029004 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-systemd-units\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029011 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-cnibin\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029028 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-node-log\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029077 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-cni-bin\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029081 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-device-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029110 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029118 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b36b8f12-981e-46a8-b422-632babc5d4a7-kubelet-config\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029134 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-etc-kubernetes\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029169 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57d5q\" (UniqueName: \"kubernetes.io/projected/67036dae-0e4a-473b-af97-e0ee20d824b8-kube-api-access-57d5q\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029174 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-etc-kubernetes\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029171 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-socket-dir-parent\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029181 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029124 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-cni-bin\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029203 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-run\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029225 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a2196112-4441-49a3-b48f-1b80bf426e29-etc-tuned\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.030280 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029252 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029269 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysctl-d\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029297 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029309 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-slash\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029319 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-var-lib-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029342 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb2c72e6-372c-41c1-ba29-d47c0b521f34-host\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029343 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-system-cni-dir\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029364 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-node-log\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029046 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd2b0c8e-8194-456d-8299-c43b1688404a-tmp-dir\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029404 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-sys-fs\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029272 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-system-cni-dir\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029425 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-systemd-units\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029456 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-system-cni-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029459 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029481 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-netns\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029491 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-run\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.029531 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:19:29.031093 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029546 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b36b8f12-981e-46a8-b422-632babc5d4a7-dbus\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.029579 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret podName:b36b8f12-981e-46a8-b422-632babc5d4a7 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:29.529564135 +0000 UTC m=+2.087121638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret") pod "global-pull-secret-syncer-lt2pz" (UID: "b36b8f12-981e-46a8-b422-632babc5d4a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029531 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-netns\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029594 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-systemd\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029600 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-system-cni-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029630 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-hostroot\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029643 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029657 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-conf-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-kubernetes\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029684 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-hostroot\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029707 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-lib-modules\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029719 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-conf-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029731 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029756 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67036dae-0e4a-473b-af97-e0ee20d824b8-cni-binary-copy\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029782 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-etc-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029806 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-cni-netd\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.031793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029854 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-kubelet\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029898 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-kubelet\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029899 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029946 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029953 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-etc-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.029962 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-daemon-config\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030010 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-kubernetes\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030014 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qnf9\" (UniqueName: \"kubernetes.io/projected/a2196112-4441-49a3-b48f-1b80bf426e29-kube-api-access-4qnf9\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030108 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-systemd\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030135 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovnkube-script-lib\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030177 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-k8s-cni-cncf-io\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030011 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030045 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-cni-netd\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030227 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-run-netns\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030270 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-systemd\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030334 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-run-netns\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.032599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030382 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-k8s-cni-cncf-io\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.033239 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.030579 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-lib-modules\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.033239 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.031328 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-env-overrides\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.033239 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.031347 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.033239 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.031355 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb2c72e6-372c-41c1-ba29-d47c0b521f34-serviceca\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:19:29.033239 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.031426 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e07b376-b6a8-4d05-b6f4-705695c0e92b-iptables-alerter-script\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:19:29.033239 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.031537 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67036dae-0e4a-473b-af97-e0ee20d824b8-cni-binary-copy\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.033239 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.031661 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-daemon-config\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.033239 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.032041 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovnkube-config\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.033239 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.032187 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovnkube-script-lib\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.033501 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.033321 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a2196112-4441-49a3-b48f-1b80bf426e29-etc-tuned\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.033501 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.033333 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2196112-4441-49a3-b48f-1b80bf426e29-tmp\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.033557 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.033526 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30634e3d-5f01-4b9b-88ad-03893a2b444e-agent-certs\") pod \"konnectivity-agent-h8rzc\" (UID: \"30634e3d-5f01-4b9b-88ad-03893a2b444e\") " pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:19:29.036115 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.034966 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovn-node-metrics-cert\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.036394 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.036131 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:19:29.052496 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.052472 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rbfc\" (UniqueName: \"kubernetes.io/projected/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-kube-api-access-2rbfc\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.053375 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.053347 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:19:29.053479 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.053378 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:19:29.053479 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.053393 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8skbh for pod openshift-network-diagnostics/network-check-target-mv4lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:19:29.053479 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.053456 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh podName:d90a91c8-f7a6-4526-a13c-ce484713b6b5 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:29.553438477 +0000 UTC m=+2.110996002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8skbh" (UniqueName: "kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh") pod "network-check-target-mv4lc" (UID: "d90a91c8-f7a6-4526-a13c-ce484713b6b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:19:29.055118 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.055089 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99d2h\" (UniqueName: \"kubernetes.io/projected/fb2c72e6-372c-41c1-ba29-d47c0b521f34-kube-api-access-99d2h\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:19:29.055256 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.055230 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblkc\" (UniqueName: \"kubernetes.io/projected/cd2b0c8e-8194-456d-8299-c43b1688404a-kube-api-access-cblkc\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:19:29.055329 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.055269 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6c2l\" (UniqueName: \"kubernetes.io/projected/ddebdc11-43b3-4787-a3ff-bf95150f8d27-kube-api-access-s6c2l\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:29.055602 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.055583 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkz5f\" (UniqueName: \"kubernetes.io/projected/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-kube-api-access-qkz5f\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.055722 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.055701 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jd74\" (UniqueName: \"kubernetes.io/projected/3e07b376-b6a8-4d05-b6f4-705695c0e92b-kube-api-access-2jd74\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:19:29.056100 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.056081 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qnf9\" (UniqueName: \"kubernetes.io/projected/a2196112-4441-49a3-b48f-1b80bf426e29-kube-api-access-4qnf9\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.056246 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.056227 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk6g4\" (UniqueName: \"kubernetes.io/projected/81a6d8ed-431a-4b9b-bd73-d13b01208249-kube-api-access-bk6g4\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.056621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.056605 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57d5q\" (UniqueName: \"kubernetes.io/projected/67036dae-0e4a-473b-af97-e0ee20d824b8-kube-api-access-57d5q\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.185492 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.185456 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:19:29.194841 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:29.194817 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2c72e6_372c_41c1_ba29_d47c0b521f34.slice/crio-37d8bfdd628d25d8061da4f3a87c1a8b41de644e58a636ec0551ec9cb59ac51f WatchSource:0}: Error finding container 37d8bfdd628d25d8061da4f3a87c1a8b41de644e58a636ec0551ec9cb59ac51f: Status 404 returned error can't find the container with id 37d8bfdd628d25d8061da4f3a87c1a8b41de644e58a636ec0551ec9cb59ac51f Oct 30 16:19:29.198032 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.198009 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:19:29.205547 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:29.205524 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2b0c8e_8194_456d_8299_c43b1688404a.slice/crio-0fcc8b41f6f78812d8fd579ca43e70a488b6f9659ef2efa107fdef1be43b3c4d WatchSource:0}: Error finding container 0fcc8b41f6f78812d8fd579ca43e70a488b6f9659ef2efa107fdef1be43b3c4d: Status 404 returned error can't find the container with id 0fcc8b41f6f78812d8fd579ca43e70a488b6f9659ef2efa107fdef1be43b3c4d Oct 30 16:19:29.223000 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.222965 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qdzgt" Oct 30 16:19:29.230379 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:29.230358 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67036dae_0e4a_473b_af97_e0ee20d824b8.slice/crio-db4b67644dc5923ad73e9ba0e1a4aa59668646dba0c302557567627c4838b297 WatchSource:0}: Error finding container db4b67644dc5923ad73e9ba0e1a4aa59668646dba0c302557567627c4838b297: Status 404 returned error can't find the container with id db4b67644dc5923ad73e9ba0e1a4aa59668646dba0c302557567627c4838b297 Oct 30 16:19:29.241661 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.241642 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:19:29.248663 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:29.248642 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e07b376_b6a8_4d05_b6f4_705695c0e92b.slice/crio-13fe18553811e4ef9d4d2cd07996a2abb66728c7d29d2f1a67381e0009c5d1f6 WatchSource:0}: Error finding container 13fe18553811e4ef9d4d2cd07996a2abb66728c7d29d2f1a67381e0009c5d1f6: Status 404 returned error can't find the container with id 13fe18553811e4ef9d4d2cd07996a2abb66728c7d29d2f1a67381e0009c5d1f6 Oct 30 16:19:29.260062 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.260046 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:19:29.270392 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:29.270371 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b54ca32_bcd3_4baa_bd63_0ee35bb5d6e2.slice/crio-e6b17f375c7f7270b0f267083307e091303bd28d80da37550d26258867925c02 WatchSource:0}: Error finding container e6b17f375c7f7270b0f267083307e091303bd28d80da37550d26258867925c02: Status 404 returned error can't find the container with id e6b17f375c7f7270b0f267083307e091303bd28d80da37550d26258867925c02 Oct 30 16:19:29.273540 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.273519 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:19:29.277449 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.277429 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:29.280774 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:29.280753 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2196112_4441_49a3_b48f_1b80bf426e29.slice/crio-1079f761db87e6123f4e4df1f7c10044a8cbde7687ea399696afd7cb28d30e16 WatchSource:0}: Error finding container 1079f761db87e6123f4e4df1f7c10044a8cbde7687ea399696afd7cb28d30e16: Status 404 returned error can't find the container with id 1079f761db87e6123f4e4df1f7c10044a8cbde7687ea399696afd7cb28d30e16 Oct 30 16:19:29.286619 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:29.286598 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b13dfe3_9da5_4be8_9051_b1bbff1b083b.slice/crio-b75f55b1eabe52dfe9a1d14564c828142c3dae665cf2a30ff0180f333b3d3ab7 WatchSource:0}: Error finding container b75f55b1eabe52dfe9a1d14564c828142c3dae665cf2a30ff0180f333b3d3ab7: Status 404 returned error can't find the container with id b75f55b1eabe52dfe9a1d14564c828142c3dae665cf2a30ff0180f333b3d3ab7 Oct 30 16:19:29.291800 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.291781 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:19:29.292927 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.292910 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:19:29.301460 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:29.301436 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30634e3d_5f01_4b9b_88ad_03893a2b444e.slice/crio-948f3d754808f828efa22bc27a3d7c31b701d85424db6ceb405de474567aaa84 WatchSource:0}: Error finding container 948f3d754808f828efa22bc27a3d7c31b701d85424db6ceb405de474567aaa84: Status 404 returned error can't find the container with id 948f3d754808f828efa22bc27a3d7c31b701d85424db6ceb405de474567aaa84 Oct 30 16:19:29.302522 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:19:29.302503 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a6d8ed_431a_4b9b_bd73_d13b01208249.slice/crio-22376cc752f611717ad034681e2aeb5629a1f1848734d9520754118aea8e962a WatchSource:0}: Error finding container 22376cc752f611717ad034681e2aeb5629a1f1848734d9520754118aea8e962a: Status 404 returned error can't find the container with id 22376cc752f611717ad034681e2aeb5629a1f1848734d9520754118aea8e962a Oct 30 16:19:29.533408 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.533274 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:29.533408 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.533348 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:29.533620 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.533453 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:19:29.533620 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.533527 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret podName:b36b8f12-981e-46a8-b422-632babc5d4a7 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:30.533510695 +0000 UTC m=+3.091068202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret") pod "global-pull-secret-syncer-lt2pz" (UID: "b36b8f12-981e-46a8-b422-632babc5d4a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:19:29.533620 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.533453 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:19:29.533770 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.533633 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs podName:ddebdc11-43b3-4787-a3ff-bf95150f8d27 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:30.533606471 +0000 UTC m=+3.091163987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs") pod "network-metrics-daemon-4856h" (UID: "ddebdc11-43b3-4787-a3ff-bf95150f8d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:19:29.572653 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.572621 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:19:29.634310 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.634277 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skbh\" (UniqueName: \"kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh\") pod \"network-check-target-mv4lc\" (UID: \"d90a91c8-f7a6-4526-a13c-ce484713b6b5\") " pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:29.634516 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.634462 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:19:29.634516 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.634490 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:19:29.634516 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.634505 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8skbh for pod openshift-network-diagnostics/network-check-target-mv4lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:19:29.634675 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.634565 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh podName:d90a91c8-f7a6-4526-a13c-ce484713b6b5 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:30.634546094 +0000 UTC m=+3.192103621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8skbh" (UniqueName: "kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh") pod "network-check-target-mv4lc" (UID: "d90a91c8-f7a6-4526-a13c-ce484713b6b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:19:29.801738 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.801665 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:19:29.891068 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.891006 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:14:28 +0000 UTC" deadline="2028-05-27 23:09:16.171747754 +0000 UTC" Oct 30 16:19:29.891068 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.891064 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13806h49m46.280687705s" Oct 30 16:19:29.928056 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.928027 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:19:29.968235 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:29.968182 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:29.968387 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:29.968306 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:30.002960 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.002894 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" event={"ID":"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2","Type":"ContainerStarted","Data":"e6b17f375c7f7270b0f267083307e091303bd28d80da37550d26258867925c02"} Oct 30 16:19:30.006697 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.006618 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c2tkd" event={"ID":"3e07b376-b6a8-4d05-b6f4-705695c0e92b","Type":"ContainerStarted","Data":"13fe18553811e4ef9d4d2cd07996a2abb66728c7d29d2f1a67381e0009c5d1f6"} Oct 30 16:19:30.015294 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.015206 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7rnq2" event={"ID":"cd2b0c8e-8194-456d-8299-c43b1688404a","Type":"ContainerStarted","Data":"0fcc8b41f6f78812d8fd579ca43e70a488b6f9659ef2efa107fdef1be43b3c4d"} Oct 30 16:19:30.035089 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.035053 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-46t9k" event={"ID":"fb2c72e6-372c-41c1-ba29-d47c0b521f34","Type":"ContainerStarted","Data":"37d8bfdd628d25d8061da4f3a87c1a8b41de644e58a636ec0551ec9cb59ac51f"} Oct 30 16:19:30.051032 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.050998 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" event={"ID":"a8e5ad7f7f3345c1093a31c3b5fc9d86","Type":"ContainerStarted","Data":"5aeb3fbd398c51d8dc64868da92878d26a596a7beb445896d953f1650f7cc0c1"} Oct 30 16:19:30.057148 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.057072 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" event={"ID":"823ffa96cf0d2439ccc4c6edac7c16b5","Type":"ContainerStarted","Data":"62e319000ca88b93d246d429f68a234857e121cbee4eb5504346e3d0ba39c0b8"} Oct 30 16:19:30.074633 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.074595 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" event={"ID":"4b13dfe3-9da5-4be8-9051-b1bbff1b083b","Type":"ContainerStarted","Data":"b75f55b1eabe52dfe9a1d14564c828142c3dae665cf2a30ff0180f333b3d3ab7"} Oct 30 16:19:30.092087 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.092054 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4g45x" event={"ID":"a2196112-4441-49a3-b48f-1b80bf426e29","Type":"ContainerStarted","Data":"1079f761db87e6123f4e4df1f7c10044a8cbde7687ea399696afd7cb28d30e16"} Oct 30 16:19:30.121287 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.121250 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qdzgt" event={"ID":"67036dae-0e4a-473b-af97-e0ee20d824b8","Type":"ContainerStarted","Data":"db4b67644dc5923ad73e9ba0e1a4aa59668646dba0c302557567627c4838b297"} Oct 30 16:19:30.147991 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.143869 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvx" event={"ID":"81a6d8ed-431a-4b9b-bd73-d13b01208249","Type":"ContainerStarted","Data":"22376cc752f611717ad034681e2aeb5629a1f1848734d9520754118aea8e962a"} Oct 30 16:19:30.162587 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.162546 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h8rzc" event={"ID":"30634e3d-5f01-4b9b-88ad-03893a2b444e","Type":"ContainerStarted","Data":"948f3d754808f828efa22bc27a3d7c31b701d85424db6ceb405de474567aaa84"} Oct 30 16:19:30.545557 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.545519 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:30.545730 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.545604 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:30.545809 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:30.545744 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:19:30.545861 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:30.545810 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs podName:ddebdc11-43b3-4787-a3ff-bf95150f8d27 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:32.545788629 +0000 UTC m=+5.103346138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs") pod "network-metrics-daemon-4856h" (UID: "ddebdc11-43b3-4787-a3ff-bf95150f8d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:19:30.546358 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:30.546339 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:19:30.546463 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:30.546395 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret podName:b36b8f12-981e-46a8-b422-632babc5d4a7 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:32.546379637 +0000 UTC m=+5.103937145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret") pod "global-pull-secret-syncer-lt2pz" (UID: "b36b8f12-981e-46a8-b422-632babc5d4a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:19:30.646663 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.646614 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skbh\" (UniqueName: \"kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh\") pod \"network-check-target-mv4lc\" (UID: \"d90a91c8-f7a6-4526-a13c-ce484713b6b5\") " pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:30.646834 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:30.646801 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:19:30.646834 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:30.646819 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:19:30.646834 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:30.646832 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8skbh for pod openshift-network-diagnostics/network-check-target-mv4lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:19:30.646999 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:30.646888 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh podName:d90a91c8-f7a6-4526-a13c-ce484713b6b5 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:32.646870696 +0000 UTC m=+5.204428214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8skbh" (UniqueName: "kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh") pod "network-check-target-mv4lc" (UID: "d90a91c8-f7a6-4526-a13c-ce484713b6b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:19:30.892126 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.892037 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:14:28 +0000 UTC" deadline="2028-04-26 01:14:06.217288455 +0000 UTC" Oct 30 16:19:30.892126 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.892082 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13040h54m35.325211458s" Oct 30 16:19:30.965679 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.965649 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:30.965856 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:30.965747 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:30.965856 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:30.965808 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:30.965987 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:30.965887 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:31.967713 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:31.967168 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:31.967713 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:31.967311 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:32.562580 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:32.562174 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:32.562580 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:32.562253 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:32.562580 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:32.562377 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:19:32.562580 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:32.562430 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs podName:ddebdc11-43b3-4787-a3ff-bf95150f8d27 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:36.562413875 +0000 UTC m=+9.119971403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs") pod "network-metrics-daemon-4856h" (UID: "ddebdc11-43b3-4787-a3ff-bf95150f8d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:19:32.562580 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:32.562495 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:19:32.562580 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:32.562529 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret podName:b36b8f12-981e-46a8-b422-632babc5d4a7 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:36.56251872 +0000 UTC m=+9.120076237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret") pod "global-pull-secret-syncer-lt2pz" (UID: "b36b8f12-981e-46a8-b422-632babc5d4a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:19:32.662955 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:32.662925 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skbh\" (UniqueName: \"kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh\") pod \"network-check-target-mv4lc\" (UID: \"d90a91c8-f7a6-4526-a13c-ce484713b6b5\") " pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:32.663132 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:32.663063 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:19:32.663132 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:32.663078 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:19:32.663132 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:32.663086 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8skbh for pod openshift-network-diagnostics/network-check-target-mv4lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:19:32.663132 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:32.663131 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh podName:d90a91c8-f7a6-4526-a13c-ce484713b6b5 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:36.663115264 +0000 UTC m=+9.220672793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8skbh" (UniqueName: "kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh") pod "network-check-target-mv4lc" (UID: "d90a91c8-f7a6-4526-a13c-ce484713b6b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:19:32.965858 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:32.965467 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:32.965858 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:32.965706 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:32.965858 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:32.965484 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:32.966040 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:32.965918 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:33.179548 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:33.179436 2583 generic.go:356] "Generic (PLEG): container finished" podID="823ffa96cf0d2439ccc4c6edac7c16b5" containerID="af1e1b931cbd4a97cce4f14bd0bfb9246e1a7f2e5a66135c1b6dba05de613bf6" exitCode=0 Oct 30 16:19:33.182566 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:33.182157 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" event={"ID":"823ffa96cf0d2439ccc4c6edac7c16b5","Type":"ContainerDied","Data":"af1e1b931cbd4a97cce4f14bd0bfb9246e1a7f2e5a66135c1b6dba05de613bf6"} Oct 30 16:19:33.187158 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:33.187124 2583 generic.go:356] "Generic (PLEG): container finished" podID="81a6d8ed-431a-4b9b-bd73-d13b01208249" containerID="aee26331736e802e4da5198b40f5a7a5a961a85750f6017a2724089e33641e12" exitCode=0 Oct 30 16:19:33.187267 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:33.187176 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvx" event={"ID":"81a6d8ed-431a-4b9b-bd73-d13b01208249","Type":"ContainerDied","Data":"aee26331736e802e4da5198b40f5a7a5a961a85750f6017a2724089e33641e12"} Oct 30 16:19:33.965959 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:33.965923 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:33.966154 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:33.966068 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:34.196660 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:34.196619 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h8rzc" event={"ID":"30634e3d-5f01-4b9b-88ad-03893a2b444e","Type":"ContainerStarted","Data":"1ed8e51ba1b2ca18da5be84aad88b4ed5fc1cab3c44e81a105b625bff24cab35"} Oct 30 16:19:34.201643 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:34.201097 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" event={"ID":"a8e5ad7f7f3345c1093a31c3b5fc9d86","Type":"ContainerStarted","Data":"1e52594d2108310a8320eb6a6b30fcebba9d8dbc3fdb4cd6bed225dee14d3ec5"} Oct 30 16:19:34.204380 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:34.203893 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" event={"ID":"823ffa96cf0d2439ccc4c6edac7c16b5","Type":"ContainerStarted","Data":"03dd121c72aab9cd33a9ad8730fe5e799e78a5e5640471b84eeb5c0a1105871d"} Oct 30 16:19:34.241166 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:34.240684 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h8rzc" podStartSLOduration=1.9363072749999999 podStartE2EDuration="6.2406637s" totalImagesPullingTime="4.304356425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:19:34.239719116 +0000 UTC m=+6.797276668" watchObservedRunningTime="2026-10-30 16:19:34.2406637 +0000 UTC m=+6.798221226" Oct 30 16:19:34.325983 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:34.325390 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" podStartSLOduration=6.325368842 podStartE2EDuration="6.325368842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:19:34.281197522 +0000 UTC m=+6.838755047" watchObservedRunningTime="2026-10-30 16:19:34.325368842 +0000 UTC m=+6.882926369" Oct 30 16:19:34.325983 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:34.325826 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" podStartSLOduration=6.325818617 podStartE2EDuration="6.325818617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:19:34.324688195 +0000 UTC m=+6.882245735" watchObservedRunningTime="2026-10-30 16:19:34.325818617 +0000 UTC m=+6.883376143" Oct 30 16:19:34.965791 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:34.965755 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:34.966028 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:34.965897 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:34.966332 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:34.966311 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:34.966441 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:34.966421 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:35.057374 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:35.057343 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:19:35.057844 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:35.057826 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:19:35.214706 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:35.213905 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-46t9k" event={"ID":"fb2c72e6-372c-41c1-ba29-d47c0b521f34","Type":"ContainerStarted","Data":"dd7fcccce4982648aa364dd5dd166010782ba0732c698ac52645d06bb718723a"} Oct 30 16:19:35.968518 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:35.968468 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:35.968704 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:35.968592 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:36.218203 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:36.217417 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:19:36.218203 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:36.217828 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" event={"ID":"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2","Type":"ContainerStarted","Data":"080d1ff8b3912ff298792aa9ad5c7944b0d3aada379f356d526f851e1b15bc5a"} Oct 30 16:19:36.599338 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:36.599301 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:36.599541 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:36.599380 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:36.599541 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:36.599505 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:19:36.599654 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:36.599569 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs podName:ddebdc11-43b3-4787-a3ff-bf95150f8d27 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:44.5995496 +0000 UTC m=+17.157107106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs") pod "network-metrics-daemon-4856h" (UID: "ddebdc11-43b3-4787-a3ff-bf95150f8d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:19:36.600043 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:36.600022 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:19:36.600144 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:36.600080 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret podName:b36b8f12-981e-46a8-b422-632babc5d4a7 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:44.600063843 +0000 UTC m=+17.157621363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret") pod "global-pull-secret-syncer-lt2pz" (UID: "b36b8f12-981e-46a8-b422-632babc5d4a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:19:36.700586 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:36.700447 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skbh\" (UniqueName: \"kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh\") pod \"network-check-target-mv4lc\" (UID: \"d90a91c8-f7a6-4526-a13c-ce484713b6b5\") " pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:36.700744 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:36.700648 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:19:36.700744 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:36.700672 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:19:36.700744 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:36.700687 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8skbh for pod openshift-network-diagnostics/network-check-target-mv4lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:19:36.700888 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:36.700753 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh podName:d90a91c8-f7a6-4526-a13c-ce484713b6b5 nodeName:}" failed. No retries permitted until 2026-10-30 16:19:44.70073398 +0000 UTC m=+17.258291498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8skbh" (UniqueName: "kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh") pod "network-check-target-mv4lc" (UID: "d90a91c8-f7a6-4526-a13c-ce484713b6b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:19:36.885388 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:36.885357 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:19:36.893508 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:36.893415 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:19:36.885386041Z","UUID":"6971403c-5c4f-4f78-970d-fc85dfd004c2","Handler":null,"Name":"","Endpoint":""} Oct 30 16:19:36.894740 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:36.894716 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:19:36.894740 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:36.894742 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:19:36.966062 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:36.965630 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:36.966062 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:36.965810 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:36.966283 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:36.966223 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:36.966337 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:36.966323 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:37.222323 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:37.222237 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" event={"ID":"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2","Type":"ContainerStarted","Data":"15ffca6b3bd91f97fc5218ef63e98650528a6354fc45c87b958c494db4e48346"} Oct 30 16:19:37.966506 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:37.966466 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:37.966676 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:37.966598 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:38.227657 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:38.227432 2583 generic.go:356] "Generic (PLEG): container finished" podID="81a6d8ed-431a-4b9b-bd73-d13b01208249" containerID="927ba85ddd325374760076133f8f1fb6a3f8e49cc13a875710cf66659d9ccb60" exitCode=0 Oct 30 16:19:38.228048 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:38.227702 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvx" event={"ID":"81a6d8ed-431a-4b9b-bd73-d13b01208249","Type":"ContainerDied","Data":"927ba85ddd325374760076133f8f1fb6a3f8e49cc13a875710cf66659d9ccb60"} Oct 30 16:19:38.230460 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:38.230434 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" event={"ID":"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2","Type":"ContainerStarted","Data":"23d29e7cce4eacee7c8e9712a8bfd9aa48025181ce272a8114d0132b75e74da3"} Oct 30 16:19:38.235026 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:38.234651 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7rnq2" event={"ID":"cd2b0c8e-8194-456d-8299-c43b1688404a","Type":"ContainerStarted","Data":"8472d2fd3a3091a7a1d50bcf267ba630edec35da7bc6f3d06eefcaf4880f87e9"} Oct 30 16:19:38.236502 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:38.236478 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qdzgt" event={"ID":"67036dae-0e4a-473b-af97-e0ee20d824b8","Type":"ContainerStarted","Data":"5580889d682056633871f06662b41cfac8164efcea54c4c229ef244eb7b48a84"} Oct 30 16:19:38.278588 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:38.278536 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-46t9k" podStartSLOduration=5.044957654 podStartE2EDuration="10.27852276s" totalImagesPullingTime="5.233565106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:19:35.266108092 +0000 UTC m=+7.823665617" watchObservedRunningTime="2026-10-30 16:19:38.27852276 +0000 UTC m=+10.836080284" Oct 30 16:19:38.372911 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:38.372806 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qdzgt" podStartSLOduration=2.201149211 podStartE2EDuration="10.3727858s" totalImagesPullingTime="8.171636589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:19:38.37233258 +0000 UTC m=+10.929890109" watchObservedRunningTime="2026-10-30 16:19:38.3727858 +0000 UTC m=+10.930343328" Oct 30 16:19:38.373119 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:38.373084 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" podStartSLOduration=1.964378856 podStartE2EDuration="10.373074275s" totalImagesPullingTime="8.408695419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:19:38.325802911 +0000 UTC m=+10.883360452" watchObservedRunningTime="2026-10-30 16:19:38.373074275 +0000 UTC m=+10.930631807" Oct 30 16:19:38.416060 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:38.415995 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7rnq2" podStartSLOduration=2.2669020140000002 podStartE2EDuration="10.415958779s" totalImagesPullingTime="8.149056765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:19:38.415819075 +0000 UTC m=+10.973376601" watchObservedRunningTime="2026-10-30 16:19:38.415958779 +0000 UTC m=+10.973516305" Oct 30 16:19:38.966014 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:38.965957 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:38.966182 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:38.966015 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:38.966182 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:38.966141 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:38.966304 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:38.966278 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:39.240861 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:39.240829 2583 generic.go:356] "Generic (PLEG): container finished" podID="81a6d8ed-431a-4b9b-bd73-d13b01208249" containerID="5c13a014aed4350aa3eecd49f4ad82f35b7eb4bae3ae8dd75fb42ae63e0662db" exitCode=0 Oct 30 16:19:39.241313 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:39.241153 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvx" event={"ID":"81a6d8ed-431a-4b9b-bd73-d13b01208249","Type":"ContainerDied","Data":"5c13a014aed4350aa3eecd49f4ad82f35b7eb4bae3ae8dd75fb42ae63e0662db"} Oct 30 16:19:39.820042 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:39.820017 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7rnq2_cd2b0c8e-8194-456d-8299-c43b1688404a/dns-node-resolver/0.log" Oct 30 16:19:39.965869 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:39.965734 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:39.965993 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:39.965857 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:40.244316 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:40.244276 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4g45x" event={"ID":"a2196112-4441-49a3-b48f-1b80bf426e29","Type":"ContainerStarted","Data":"ee7ae105f88c7c93bd250bbe91ff60bf82bb5a1d73ca6747a5b7c3504b6c1010"} Oct 30 16:19:40.247036 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:40.247005 2583 generic.go:356] "Generic (PLEG): container finished" podID="81a6d8ed-431a-4b9b-bd73-d13b01208249" containerID="8a6b138d39000d13a0ef795b896f63fe0d41b75419b9d89d4ab60f8bcbbcc168" exitCode=0 Oct 30 16:19:40.247168 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:40.247102 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvx" event={"ID":"81a6d8ed-431a-4b9b-bd73-d13b01208249","Type":"ContainerDied","Data":"8a6b138d39000d13a0ef795b896f63fe0d41b75419b9d89d4ab60f8bcbbcc168"} Oct 30 16:19:40.248650 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:40.248625 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c2tkd" event={"ID":"3e07b376-b6a8-4d05-b6f4-705695c0e92b","Type":"ContainerStarted","Data":"8b7c6c55189d7602536ca614b852d9cffa9a6c87575cc9ae2ccef93fd49fb933"} Oct 30 16:19:40.291034 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:40.290965 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4g45x" podStartSLOduration=2.223254364 podStartE2EDuration="12.290953233s" totalImagesPullingTime="10.067698869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:19:40.290449927 +0000 UTC m=+12.848007456" watchObservedRunningTime="2026-10-30 16:19:40.290953233 +0000 UTC m=+12.848510757" Oct 30 16:19:40.333677 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:40.333608 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c2tkd" podStartSLOduration=3.877258075 podStartE2EDuration="12.33359326s" totalImagesPullingTime="8.456335185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:19:40.332936398 +0000 UTC m=+12.890493924" watchObservedRunningTime="2026-10-30 16:19:40.33359326 +0000 UTC m=+12.891150785" Oct 30 16:19:40.729861 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:40.729788 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-46t9k_fb2c72e6-372c-41c1-ba29-d47c0b521f34/node-ca/0.log" Oct 30 16:19:40.966351 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:40.966311 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:40.966522 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:40.966354 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:40.966522 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:40.966439 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:40.966648 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:40.966559 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:41.204492 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:41.204457 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8rzc_30634e3d-5f01-4b9b-88ad-03893a2b444e/konnectivity-agent/0.log" limit=1024 Oct 30 16:19:41.543470 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:41.543436 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8rzc_30634e3d-5f01-4b9b-88ad-03893a2b444e/konnectivity-agent/0.log" limit=1024 Oct 30 16:19:41.940326 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:41.940294 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8rzc_30634e3d-5f01-4b9b-88ad-03893a2b444e/konnectivity-agent/0.log" limit=1024 Oct 30 16:19:41.965406 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:41.965376 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:41.965567 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:41.965517 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:42.109625 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:42.109529 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h8rzc_30634e3d-5f01-4b9b-88ad-03893a2b444e/konnectivity-agent/0.log" limit=1024 Oct 30 16:19:42.257424 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:42.257382 2583 generic.go:356] "Generic (PLEG): container finished" podID="81a6d8ed-431a-4b9b-bd73-d13b01208249" containerID="023946bfd7db7d3cfbd325d974b363ade09bfe7d953089c4e817e48ec9b4f412" exitCode=0 Oct 30 16:19:42.257424 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:42.257427 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvx" event={"ID":"81a6d8ed-431a-4b9b-bd73-d13b01208249","Type":"ContainerDied","Data":"023946bfd7db7d3cfbd325d974b363ade09bfe7d953089c4e817e48ec9b4f412"} Oct 30 16:19:42.966668 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:42.966430 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:42.967342 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:42.966468 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:42.967342 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:42.966779 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:42.967342 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:42.966873 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:43.263765 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:43.263731 2583 generic.go:356] "Generic (PLEG): container finished" podID="81a6d8ed-431a-4b9b-bd73-d13b01208249" containerID="23e798cd3d296ca0cf00b0e8537d106877e72078d221be67ca8a78412b30e120" exitCode=0 Oct 30 16:19:43.263930 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:43.263772 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvx" event={"ID":"81a6d8ed-431a-4b9b-bd73-d13b01208249","Type":"ContainerDied","Data":"23e798cd3d296ca0cf00b0e8537d106877e72078d221be67ca8a78412b30e120"} Oct 30 16:19:43.965378 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:43.965335 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:43.965550 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:43.965444 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:44.269273 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:44.269241 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vcfvx" event={"ID":"81a6d8ed-431a-4b9b-bd73-d13b01208249","Type":"ContainerStarted","Data":"fbba1e239f8a2bb291fa668beaa97cc58270298c431ed84fccb0912f6d324069"} Oct 30 16:19:44.320110 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:44.320074 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vcfvx" podStartSLOduration=6.311878326 podStartE2EDuration="16.320059456s" totalImagesPullingTime="10.00818113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:19:44.31991272 +0000 UTC m=+16.877470293" watchObservedRunningTime="2026-10-30 16:19:44.320059456 +0000 UTC m=+16.877616972" Oct 30 16:19:44.687121 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:44.687089 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:44.687309 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:44.687142 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:44.687309 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:44.687251 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:19:44.687309 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:44.687262 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:19:44.687309 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:44.687302 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret podName:b36b8f12-981e-46a8-b422-632babc5d4a7 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:00.6872855 +0000 UTC m=+33.244843016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret") pod "global-pull-secret-syncer-lt2pz" (UID: "b36b8f12-981e-46a8-b422-632babc5d4a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:19:44.687459 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:44.687319 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs podName:ddebdc11-43b3-4787-a3ff-bf95150f8d27 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:00.687311846 +0000 UTC m=+33.244869349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs") pod "network-metrics-daemon-4856h" (UID: "ddebdc11-43b3-4787-a3ff-bf95150f8d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:19:44.787793 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:44.787755 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skbh\" (UniqueName: \"kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh\") pod \"network-check-target-mv4lc\" (UID: \"d90a91c8-f7a6-4526-a13c-ce484713b6b5\") " pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:44.787964 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:44.787902 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:19:44.787964 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:44.787917 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:19:44.787964 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:44.787927 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8skbh for pod openshift-network-diagnostics/network-check-target-mv4lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:19:44.788064 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:44.787994 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh podName:d90a91c8-f7a6-4526-a13c-ce484713b6b5 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:00.787960725 +0000 UTC m=+33.345518228 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8skbh" (UniqueName: "kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh") pod "network-check-target-mv4lc" (UID: "d90a91c8-f7a6-4526-a13c-ce484713b6b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:19:44.965396 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:44.965364 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:44.965577 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:44.965497 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:44.965577 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:44.965542 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:44.965655 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:44.965615 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:45.274077 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:45.274054 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:19:45.274492 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:45.274380 2583 generic.go:356] "Generic (PLEG): container finished" podID="4b13dfe3-9da5-4be8-9051-b1bbff1b083b" containerID="01a36d92f8a460b4854691eaefe1f298a98470c76e2bdce84461014e50429425" exitCode=1 Oct 30 16:19:45.274492 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:45.274465 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" event={"ID":"4b13dfe3-9da5-4be8-9051-b1bbff1b083b","Type":"ContainerStarted","Data":"025c5f3b62e4e080a3d03567aca1cb05fdd0a54554c0fa70ca5edac79fb1ed13"} Oct 30 16:19:45.274608 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:45.274550 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" event={"ID":"4b13dfe3-9da5-4be8-9051-b1bbff1b083b","Type":"ContainerStarted","Data":"07ffbb14b359925dddeeb694fc80a24336f6702de01ab90976ac9248a2435e5c"} Oct 30 16:19:45.274608 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:45.274601 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" event={"ID":"4b13dfe3-9da5-4be8-9051-b1bbff1b083b","Type":"ContainerStarted","Data":"c378df9c950bb6d5e4d02868d70a60ff100ac55c4ed73fc9667fba044509029d"} Oct 30 16:19:45.274709 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:45.274653 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" event={"ID":"4b13dfe3-9da5-4be8-9051-b1bbff1b083b","Type":"ContainerStarted","Data":"5d37cbbbcb7a9759d6d7fcf5f90870cc381a2388c46d19ecd38add0e84d346c6"} Oct 30 16:19:45.274759 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:45.274706 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" event={"ID":"4b13dfe3-9da5-4be8-9051-b1bbff1b083b","Type":"ContainerDied","Data":"01a36d92f8a460b4854691eaefe1f298a98470c76e2bdce84461014e50429425"} Oct 30 16:19:45.274811 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:45.274756 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" event={"ID":"4b13dfe3-9da5-4be8-9051-b1bbff1b083b","Type":"ContainerStarted","Data":"45f4c5eb03d1f718b6411f5e002a01c536842e9e1e5a89d825d1cf2d257e63ca"} Oct 30 16:19:45.966253 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:45.966218 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:45.966417 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:45.966344 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:46.965690 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:46.965602 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:46.965690 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:46.965655 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:46.966216 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:46.965722 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:46.966216 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:46.965831 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:47.280936 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:47.280912 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:19:47.281273 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:47.281252 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" event={"ID":"4b13dfe3-9da5-4be8-9051-b1bbff1b083b","Type":"ContainerStarted","Data":"f89a3c52a52d58e6bb141b5133855698605bc0563affc8fd1c4d46e95572f474"} Oct 30 16:19:47.966291 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:47.966266 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:47.966736 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:47.966344 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:48.965932 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:48.965896 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:48.965932 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:48.965925 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:48.966175 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:48.966051 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:48.966175 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:48.966167 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:49.288365 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:49.288339 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:19:49.289003 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:49.288660 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" event={"ID":"4b13dfe3-9da5-4be8-9051-b1bbff1b083b","Type":"ContainerStarted","Data":"20fc6ec0b1b4dce54d70e42f5a970fb1dc848fccc753b62ca1aa8d350b5ddbe8"} Oct 30 16:19:49.289003 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:49.288919 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:49.289003 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:49.289002 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:49.289167 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:49.289015 2583 scope.go:122] "RemoveContainer" containerID="01a36d92f8a460b4854691eaefe1f298a98470c76e2bdce84461014e50429425" Oct 30 16:19:49.304843 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:49.304642 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:49.966045 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:49.966019 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:49.966179 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:49.966118 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:50.242675 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:50.242646 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:19:50.242836 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:50.242770 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:19:50.243452 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:50.243429 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:19:50.294796 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:50.294773 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:19:50.295215 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:50.295176 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" event={"ID":"4b13dfe3-9da5-4be8-9051-b1bbff1b083b","Type":"ContainerStarted","Data":"79fd73ee78d24dc09b4a61c8dbe68d1cfc4e3c5fd95d722282c8da9d889a19ff"} Oct 30 16:19:50.295610 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:50.295581 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:50.310405 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:50.310386 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:19:50.370147 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:50.370105 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" podStartSLOduration=7.425797445 podStartE2EDuration="22.370093952s" totalImagesPullingTime="14.944296507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:19:50.369835051 +0000 UTC m=+22.927392577" watchObservedRunningTime="2026-10-30 16:19:50.370093952 +0000 UTC m=+22.927651476" Oct 30 16:19:50.966006 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:50.965959 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:50.966162 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:50.966071 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:50.966162 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:50.966135 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:50.966328 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:50.966304 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:51.966215 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:51.966179 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:51.966689 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:51.966295 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:52.965893 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:52.965862 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:52.966082 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:52.965864 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:52.966082 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:52.965984 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:52.966170 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:52.966108 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:53.969017 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:53.968991 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:53.969371 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:53.969080 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:54.965693 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:54.965661 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:54.965693 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:54.965694 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:54.965934 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:54.965767 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:54.965934 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:54.965831 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:55.966223 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:55.966192 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:55.966697 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:55.966289 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:56.966187 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:56.966153 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:56.966353 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:56.966256 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:56.966353 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:56.966311 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:56.966648 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:56.966410 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:57.907293 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:57.907258 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lt2pz"] Oct 30 16:19:57.907460 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:57.907415 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:57.907546 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:57.907523 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:57.910383 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:57.910150 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mv4lc"] Oct 30 16:19:57.910383 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:57.910260 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:57.910383 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:57.910351 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:57.910944 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:57.910917 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4856h"] Oct 30 16:19:57.911078 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:57.911039 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:57.911172 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:57.911149 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:58.965840 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:58.965811 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:19:58.966233 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:58.965871 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:19:58.966233 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:58.965964 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:19:58.966233 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:58.965958 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:19:58.966233 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:58.966032 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:19:58.966233 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:19:58.966124 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:19:59.001258 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:19:59.001235 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7rnq2_cd2b0c8e-8194-456d-8299-c43b1688404a/dns-node-resolver/0.log" Oct 30 16:20:00.201459 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:00.201282 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-46t9k_fb2c72e6-372c-41c1-ba29-d47c0b521f34/node-ca/0.log" Oct 30 16:20:00.697304 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:00.697267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:00.697502 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:00.697338 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:20:00.697502 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:00.697421 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:00.697502 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:00.697450 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:00.697502 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:00.697490 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret podName:b36b8f12-981e-46a8-b422-632babc5d4a7 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:32.697473265 +0000 UTC m=+65.255030781 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret") pod "global-pull-secret-syncer-lt2pz" (UID: "b36b8f12-981e-46a8-b422-632babc5d4a7") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:00.697502 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:00.697505 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs podName:ddebdc11-43b3-4787-a3ff-bf95150f8d27 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:32.697498033 +0000 UTC m=+65.255055540 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs") pod "network-metrics-daemon-4856h" (UID: "ddebdc11-43b3-4787-a3ff-bf95150f8d27") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:00.798121 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:00.798087 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skbh\" (UniqueName: \"kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh\") pod \"network-check-target-mv4lc\" (UID: \"d90a91c8-f7a6-4526-a13c-ce484713b6b5\") " pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:20:00.798302 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:00.798267 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:00.798302 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:00.798287 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:00.798302 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:00.798297 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8skbh for pod openshift-network-diagnostics/network-check-target-mv4lc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:00.798419 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:00.798344 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh podName:d90a91c8-f7a6-4526-a13c-ce484713b6b5 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:32.798331467 +0000 UTC m=+65.355888971 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8skbh" (UniqueName: "kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh") pod "network-check-target-mv4lc" (UID: "d90a91c8-f7a6-4526-a13c-ce484713b6b5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:00.965492 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:00.965417 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:00.965648 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:00.965515 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:20:00.965648 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:00.965521 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:20:00.965648 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:00.965542 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:20:00.965648 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:00.965622 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:20:00.965848 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:00.965694 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:20:02.965954 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:02.965919 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:02.965954 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:02.965951 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:20:02.966334 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:02.965958 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:20:02.966334 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:02.966060 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt2pz" podUID="b36b8f12-981e-46a8-b422-632babc5d4a7" Oct 30 16:20:02.966334 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:02.966176 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4856h" podUID="ddebdc11-43b3-4787-a3ff-bf95150f8d27" Oct 30 16:20:02.966334 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:02.966220 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mv4lc" podUID="d90a91c8-f7a6-4526-a13c-ce484713b6b5" Oct 30 16:20:03.205780 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.205693 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeReady" Oct 30 16:20:03.205944 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.205845 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 16:20:03.269474 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.269440 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vp4m7"] Oct 30 16:20:03.292946 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.292914 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n8h27"] Oct 30 16:20:03.293120 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.293051 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.309110 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.309088 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-csxp6"] Oct 30 16:20:03.309243 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.309218 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.311465 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.311444 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:20:03.311651 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.311632 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mzplq\"" Oct 30 16:20:03.311806 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.311784 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:20:03.320963 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.320943 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vp4m7"] Oct 30 16:20:03.321078 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.321006 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n8h27"] Oct 30 16:20:03.321078 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.321032 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-csxp6"] Oct 30 16:20:03.321148 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.321098 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-csxp6" Oct 30 16:20:03.331850 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.331829 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:20:03.332041 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.332024 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:20:03.332122 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.332030 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:20:03.332122 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.332053 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-snnbg\"" Oct 30 16:20:03.332122 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.332088 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:20:03.411573 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.411547 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:20:03.411804 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.411782 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zr2xc\"" Oct 30 16:20:03.411947 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.411845 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:20:03.411947 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.411866 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:20:03.416159 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.416138 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8zmd\" (UniqueName: \"kubernetes.io/projected/791cb17f-1c21-414c-9f90-196b26aa3570-kube-api-access-p8zmd\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.416240 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.416179 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791cb17f-1c21-414c-9f90-196b26aa3570-tmp-dir\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.416240 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.416200 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca822f3-98ef-487f-a89d-1a1004beeda1-data-volume\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.416240 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.416224 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ca822f3-98ef-487f-a89d-1a1004beeda1-crio-socket\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.416330 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.416295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdx66\" (UniqueName: \"kubernetes.io/projected/9ca822f3-98ef-487f-a89d-1a1004beeda1-kube-api-access-fdx66\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.416365 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.416344 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca822f3-98ef-487f-a89d-1a1004beeda1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.416396 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.416363 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca822f3-98ef-487f-a89d-1a1004beeda1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.416426 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.416406 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/791cb17f-1c21-414c-9f90-196b26aa3570-metrics-tls\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.416457 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.416441 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/791cb17f-1c21-414c-9f90-196b26aa3570-config-volume\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.517294 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/791cb17f-1c21-414c-9f90-196b26aa3570-config-volume\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.517443 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517303 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/236ec382-dbd5-4974-9454-43b39e990c77-cert\") pod \"ingress-canary-csxp6\" (UID: \"236ec382-dbd5-4974-9454-43b39e990c77\") " pod="openshift-ingress-canary/ingress-canary-csxp6" Oct 30 16:20:03.517443 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517343 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8zmd\" (UniqueName: \"kubernetes.io/projected/791cb17f-1c21-414c-9f90-196b26aa3570-kube-api-access-p8zmd\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.517443 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517371 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln9pm\" (UniqueName: \"kubernetes.io/projected/236ec382-dbd5-4974-9454-43b39e990c77-kube-api-access-ln9pm\") pod \"ingress-canary-csxp6\" (UID: \"236ec382-dbd5-4974-9454-43b39e990c77\") " pod="openshift-ingress-canary/ingress-canary-csxp6" Oct 30 16:20:03.517443 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517409 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791cb17f-1c21-414c-9f90-196b26aa3570-tmp-dir\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.517560 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517511 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca822f3-98ef-487f-a89d-1a1004beeda1-data-volume\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.517560 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517552 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ca822f3-98ef-487f-a89d-1a1004beeda1-crio-socket\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.517632 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517579 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdx66\" (UniqueName: \"kubernetes.io/projected/9ca822f3-98ef-487f-a89d-1a1004beeda1-kube-api-access-fdx66\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.517632 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517609 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca822f3-98ef-487f-a89d-1a1004beeda1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.517713 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517634 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca822f3-98ef-487f-a89d-1a1004beeda1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.517713 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517689 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/791cb17f-1c21-414c-9f90-196b26aa3570-metrics-tls\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.517803 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517746 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791cb17f-1c21-414c-9f90-196b26aa3570-tmp-dir\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.517803 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517793 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ca822f3-98ef-487f-a89d-1a1004beeda1-crio-socket\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.517890 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517843 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca822f3-98ef-487f-a89d-1a1004beeda1-data-volume\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.517932 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.517919 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/791cb17f-1c21-414c-9f90-196b26aa3570-config-volume\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.518326 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.518310 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca822f3-98ef-487f-a89d-1a1004beeda1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.521440 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.521420 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/791cb17f-1c21-414c-9f90-196b26aa3570-metrics-tls\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.521535 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.521440 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca822f3-98ef-487f-a89d-1a1004beeda1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.543640 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.543620 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdx66\" (UniqueName: \"kubernetes.io/projected/9ca822f3-98ef-487f-a89d-1a1004beeda1-kube-api-access-fdx66\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:03.543741 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.543667 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8zmd\" (UniqueName: \"kubernetes.io/projected/791cb17f-1c21-414c-9f90-196b26aa3570-kube-api-access-p8zmd\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.618796 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.618760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln9pm\" (UniqueName: \"kubernetes.io/projected/236ec382-dbd5-4974-9454-43b39e990c77-kube-api-access-ln9pm\") pod \"ingress-canary-csxp6\" (UID: \"236ec382-dbd5-4974-9454-43b39e990c77\") " pod="openshift-ingress-canary/ingress-canary-csxp6" Oct 30 16:20:03.618959 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.618837 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/236ec382-dbd5-4974-9454-43b39e990c77-cert\") pod \"ingress-canary-csxp6\" (UID: \"236ec382-dbd5-4974-9454-43b39e990c77\") " pod="openshift-ingress-canary/ingress-canary-csxp6" Oct 30 16:20:03.621302 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.621269 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/236ec382-dbd5-4974-9454-43b39e990c77-cert\") pod \"ingress-canary-csxp6\" (UID: \"236ec382-dbd5-4974-9454-43b39e990c77\") " pod="openshift-ingress-canary/ingress-canary-csxp6" Oct 30 16:20:03.635547 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.635522 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:03.644360 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.644333 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln9pm\" (UniqueName: \"kubernetes.io/projected/236ec382-dbd5-4974-9454-43b39e990c77-kube-api-access-ln9pm\") pod \"ingress-canary-csxp6\" (UID: \"236ec382-dbd5-4974-9454-43b39e990c77\") " pod="openshift-ingress-canary/ingress-canary-csxp6" Oct 30 16:20:03.650137 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.650118 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-csxp6" Oct 30 16:20:03.688305 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:03.688287 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:04.031320 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.031250 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-csxp6"] Oct 30 16:20:04.035622 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:20:04.035588 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236ec382_dbd5_4974_9454_43b39e990c77.slice/crio-3d47927b38cdc342416f26d788c9a3569434173c31a5a4746c2ee8b7bdd30fdd WatchSource:0}: Error finding container 3d47927b38cdc342416f26d788c9a3569434173c31a5a4746c2ee8b7bdd30fdd: Status 404 returned error can't find the container with id 3d47927b38cdc342416f26d788c9a3569434173c31a5a4746c2ee8b7bdd30fdd Oct 30 16:20:04.046599 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.046579 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vp4m7"] Oct 30 16:20:04.048920 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:20:04.048897 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791cb17f_1c21_414c_9f90_196b26aa3570.slice/crio-2e8b300bb7ba02af7d2ec639960543489b56a6fe8f316660cfca5d4edf70e072 WatchSource:0}: Error finding container 2e8b300bb7ba02af7d2ec639960543489b56a6fe8f316660cfca5d4edf70e072: Status 404 returned error can't find the container with id 2e8b300bb7ba02af7d2ec639960543489b56a6fe8f316660cfca5d4edf70e072 Oct 30 16:20:04.062898 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.062875 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n8h27"] Oct 30 16:20:04.065725 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:20:04.065692 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca822f3_98ef_487f_a89d_1a1004beeda1.slice/crio-1796ff68883d583b3245e9a58383981660e62e063b39996985bf98ea72810169 WatchSource:0}: Error finding container 1796ff68883d583b3245e9a58383981660e62e063b39996985bf98ea72810169: Status 404 returned error can't find the container with id 1796ff68883d583b3245e9a58383981660e62e063b39996985bf98ea72810169 Oct 30 16:20:04.322650 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.322547 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n8h27" event={"ID":"9ca822f3-98ef-487f-a89d-1a1004beeda1","Type":"ContainerStarted","Data":"f38109b8bbee6659ed550d4c861af5c4791fb74ec45f820e57f03eb38d551f04"} Oct 30 16:20:04.322650 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.322604 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n8h27" event={"ID":"9ca822f3-98ef-487f-a89d-1a1004beeda1","Type":"ContainerStarted","Data":"1796ff68883d583b3245e9a58383981660e62e063b39996985bf98ea72810169"} Oct 30 16:20:04.323705 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.323676 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-csxp6" event={"ID":"236ec382-dbd5-4974-9454-43b39e990c77","Type":"ContainerStarted","Data":"3d47927b38cdc342416f26d788c9a3569434173c31a5a4746c2ee8b7bdd30fdd"} Oct 30 16:20:04.324792 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.324766 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vp4m7" event={"ID":"791cb17f-1c21-414c-9f90-196b26aa3570","Type":"ContainerStarted","Data":"2e8b300bb7ba02af7d2ec639960543489b56a6fe8f316660cfca5d4edf70e072"} Oct 30 16:20:04.966216 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.966180 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:20:04.966638 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.966621 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:20:04.967005 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.966965 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:04.983848 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.983727 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:20:04.984044 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.984027 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:20:04.998389 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.998365 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:20:04.998505 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.998450 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t25rz\"" Oct 30 16:20:04.998547 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.998525 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2ph4\"" Oct 30 16:20:04.998577 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:04.998567 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bhx7q\"" Oct 30 16:20:05.012540 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:05.012521 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:20:05.963766 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:05.963738 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47"] Oct 30 16:20:05.977993 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:05.977449 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:05.989266 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:05.989240 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47"] Oct 30 16:20:05.996823 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:05.996798 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 16:20:05.997383 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:05.997364 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 16:20:06.002792 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.002767 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ltrwz\"" Oct 30 16:20:06.002978 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.002957 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:20:06.003087 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.003046 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:20:06.011160 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.011140 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 16:20:06.023136 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.023108 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b9fbq"] Oct 30 16:20:06.034382 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.034353 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.037743 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.037721 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-wtmp\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.037871 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.037774 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d0efab9-17c2-4f08-9433-02670766f69c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.037871 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.037808 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7bv\" (UniqueName: \"kubernetes.io/projected/3d0efab9-17c2-4f08-9433-02670766f69c-kube-api-access-4n7bv\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.038030 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.037868 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.038030 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.038012 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwhlq\" (UniqueName: \"kubernetes.io/projected/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-kube-api-access-lwhlq\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.038130 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.038086 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0efab9-17c2-4f08-9433-02670766f69c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.038185 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.038164 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-root\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.038245 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.038199 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-textfile\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.038245 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.038231 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-sys\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.038347 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.038301 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-metrics-client-ca\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.038399 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.038346 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d0efab9-17c2-4f08-9433-02670766f69c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.038399 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.038374 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.038520 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.038410 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-tls\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.052610 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.052585 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5"] Oct 30 16:20:06.077305 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.077275 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.083003 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.082916 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5"] Oct 30 16:20:06.099524 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.099497 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 16:20:06.099524 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.099524 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 16:20:06.099743 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.099567 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fjt7p\"" Oct 30 16:20:06.099743 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.099586 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 16:20:06.099743 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.099660 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lk6j9\"" Oct 30 16:20:06.099743 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.099714 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 16:20:06.099915 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.099794 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 16:20:06.099915 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.099841 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 16:20:06.139675 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.139643 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-wtmp\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.139863 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.139692 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.139863 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.139769 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d0efab9-17c2-4f08-9433-02670766f69c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.139863 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.139792 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.139863 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.139811 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/be9a1b81-45d0-46a0-a771-84d88046f28e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.139863 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.139851 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-wtmp\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.140138 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.139856 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-textfile\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.140138 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.139910 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tn95\" (UniqueName: \"kubernetes.io/projected/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-api-access-6tn95\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.140138 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.139938 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d0efab9-17c2-4f08-9433-02670766f69c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.140138 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.139956 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.140138 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.140005 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-tls\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.140138 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.140022 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.140138 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.140072 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7bv\" (UniqueName: \"kubernetes.io/projected/3d0efab9-17c2-4f08-9433-02670766f69c-kube-api-access-4n7bv\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.140138 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.140129 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.140506 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.140159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwhlq\" (UniqueName: \"kubernetes.io/projected/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-kube-api-access-lwhlq\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.140506 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.140183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0efab9-17c2-4f08-9433-02670766f69c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.140506 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.140222 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-root\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.140506 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.140246 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9a1b81-45d0-46a0-a771-84d88046f28e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.140506 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.140269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-sys\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.140506 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.140288 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-metrics-client-ca\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.140780 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.140654 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d0efab9-17c2-4f08-9433-02670766f69c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.140855 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.140834 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-metrics-client-ca\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.141287 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.141253 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.141550 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:06.141488 2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Oct 30 16:20:06.141550 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.141518 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-root\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.141550 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:06.141553 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-tls podName:ae1acb11-b25d-43d5-b027-5edcf8e5fd1b nodeName:}" failed. No retries permitted until 2026-10-30 16:20:06.641534929 +0000 UTC m=+39.199092454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-tls") pod "node-exporter-b9fbq" (UID: "ae1acb11-b25d-43d5-b027-5edcf8e5fd1b") : secret "node-exporter-tls" not found Oct 30 16:20:06.141753 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.141595 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-sys\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.141753 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.141685 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-textfile\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.144722 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.144699 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.144909 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.144884 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d0efab9-17c2-4f08-9433-02670766f69c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.145384 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.145363 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0efab9-17c2-4f08-9433-02670766f69c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.167630 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.167598 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwhlq\" (UniqueName: \"kubernetes.io/projected/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-kube-api-access-lwhlq\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.167784 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.167763 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7bv\" (UniqueName: \"kubernetes.io/projected/3d0efab9-17c2-4f08-9433-02670766f69c-kube-api-access-4n7bv\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.240696 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.240598 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9a1b81-45d0-46a0-a771-84d88046f28e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.240696 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.240636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.240696 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.240657 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.240696 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.240683 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/be9a1b81-45d0-46a0-a771-84d88046f28e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.240897 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.240732 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tn95\" (UniqueName: \"kubernetes.io/projected/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-api-access-6tn95\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.240897 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.240777 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.241337 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.241313 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9a1b81-45d0-46a0-a771-84d88046f28e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.241428 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.241402 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/be9a1b81-45d0-46a0-a771-84d88046f28e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.241484 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.241455 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.243516 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.243498 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.243742 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.243719 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.269397 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.269343 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tn95\" (UniqueName: \"kubernetes.io/projected/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-api-access-6tn95\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.330643 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.330450 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:06.332419 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.332383 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n8h27" event={"ID":"9ca822f3-98ef-487f-a89d-1a1004beeda1","Type":"ContainerStarted","Data":"6e29d09aac5de44bdcf4880dad1c2094450d431f5abb4f8418e23b41bb31af9b"} Oct 30 16:20:06.476226 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.476191 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:06.645602 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.645406 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-tls\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.649593 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.649568 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-tls\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.678069 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.678040 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47"] Oct 30 16:20:06.681690 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:20:06.681659 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0efab9_17c2_4f08_9433_02670766f69c.slice/crio-a6af2dcb8be6646a151b1dcbf23d69433191f530b283c91882258c5aa90bbd5b WatchSource:0}: Error finding container a6af2dcb8be6646a151b1dcbf23d69433191f530b283c91882258c5aa90bbd5b: Status 404 returned error can't find the container with id a6af2dcb8be6646a151b1dcbf23d69433191f530b283c91882258c5aa90bbd5b Oct 30 16:20:06.705961 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.705937 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:06.740817 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:20:06.740771 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1acb11_b25d_43d5_b027_5edcf8e5fd1b.slice/crio-27fd84afe85bce8c2321e1b1b85b813f4c9ac089819b3c63c22e4e61bf8aea4a WatchSource:0}: Error finding container 27fd84afe85bce8c2321e1b1b85b813f4c9ac089819b3c63c22e4e61bf8aea4a: Status 404 returned error can't find the container with id 27fd84afe85bce8c2321e1b1b85b813f4c9ac089819b3c63c22e4e61bf8aea4a Oct 30 16:20:06.806852 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:06.806825 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5"] Oct 30 16:20:06.812811 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:20:06.812785 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9a1b81_45d0_46a0_a771_84d88046f28e.slice/crio-c2228af531f82f28dc3beb9a5bb32696a044e1a6f85a29a9a873997054ee9940 WatchSource:0}: Error finding container c2228af531f82f28dc3beb9a5bb32696a044e1a6f85a29a9a873997054ee9940: Status 404 returned error can't find the container with id c2228af531f82f28dc3beb9a5bb32696a044e1a6f85a29a9a873997054ee9940 Oct 30 16:20:07.337947 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:07.337911 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" event={"ID":"be9a1b81-45d0-46a0-a771-84d88046f28e","Type":"ContainerStarted","Data":"c2228af531f82f28dc3beb9a5bb32696a044e1a6f85a29a9a873997054ee9940"} Oct 30 16:20:07.340633 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:07.340428 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n8h27" event={"ID":"9ca822f3-98ef-487f-a89d-1a1004beeda1","Type":"ContainerStarted","Data":"bfa9bdfe715f7fef12e31a2e2ee78a78c5936539c7459f1e4b62fd5278bdb279"} Oct 30 16:20:07.343445 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:07.343418 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vp4m7" event={"ID":"791cb17f-1c21-414c-9f90-196b26aa3570","Type":"ContainerStarted","Data":"16260ca297d28953b89c1118a34d8c2073677fbe44480c7557520a54571bc220"} Oct 30 16:20:07.343549 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:07.343475 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vp4m7" event={"ID":"791cb17f-1c21-414c-9f90-196b26aa3570","Type":"ContainerStarted","Data":"78d03bb5d81aa98499f4f91611a2753e650a87089d5dafaacfbeeaa175e1075f"} Oct 30 16:20:07.344407 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:07.344385 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:07.347096 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:07.347048 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" event={"ID":"3d0efab9-17c2-4f08-9433-02670766f69c","Type":"ContainerStarted","Data":"a151cf22a4a35de61bf64143598245d691f0a0f778bd67dce87a6588f5a17ea3"} Oct 30 16:20:07.347190 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:07.347096 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" event={"ID":"3d0efab9-17c2-4f08-9433-02670766f69c","Type":"ContainerStarted","Data":"9a5dde7dc3d21a139b28b8f59da0ff5c3350f0d09fd955e3324255618c9881b7"} Oct 30 16:20:07.347190 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:07.347128 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" event={"ID":"3d0efab9-17c2-4f08-9433-02670766f69c","Type":"ContainerStarted","Data":"a6af2dcb8be6646a151b1dcbf23d69433191f530b283c91882258c5aa90bbd5b"} Oct 30 16:20:07.350212 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:07.350169 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-csxp6" event={"ID":"236ec382-dbd5-4974-9454-43b39e990c77","Type":"ContainerStarted","Data":"329541717d5084bc54ee02bd6505be2f9e21e5321aac97e8e4cea1db1fb72fc8"} Oct 30 16:20:07.352469 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:07.352448 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9fbq" event={"ID":"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b","Type":"ContainerStarted","Data":"27fd84afe85bce8c2321e1b1b85b813f4c9ac089819b3c63c22e4e61bf8aea4a"} Oct 30 16:20:07.392246 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:07.392163 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-vp4m7" podStartSLOduration=2.203461696 podStartE2EDuration="4.392141746s" totalImagesPullingTime="2.18868005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:07.391038362 +0000 UTC m=+39.948595888" watchObservedRunningTime="2026-10-30 16:20:07.392141746 +0000 UTC m=+39.949699324" Oct 30 16:20:07.440612 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:07.440549 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-csxp6" podStartSLOduration=1.239253361 podStartE2EDuration="4.440531186s" totalImagesPullingTime="3.201277825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:07.439411164 +0000 UTC m=+39.996968688" watchObservedRunningTime="2026-10-30 16:20:07.440531186 +0000 UTC m=+39.998088715" Oct 30 16:20:08.357116 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:08.357081 2583 generic.go:356] "Generic (PLEG): container finished" podID="ae1acb11-b25d-43d5-b027-5edcf8e5fd1b" containerID="e4fa9e4166c47be78cfa740c9def8c166c5ac41127e2c0b7a95e90221c746324" exitCode=0 Oct 30 16:20:08.357484 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:08.357110 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9fbq" event={"ID":"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b","Type":"ContainerDied","Data":"e4fa9e4166c47be78cfa740c9def8c166c5ac41127e2c0b7a95e90221c746324"} Oct 30 16:20:08.459232 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:08.459186 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n8h27" podStartSLOduration=2.498770451 podStartE2EDuration="5.459173778s" totalImagesPullingTime="2.960403327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:08.458962521 +0000 UTC m=+41.016520046" watchObservedRunningTime="2026-10-30 16:20:08.459173778 +0000 UTC m=+41.016731303" Oct 30 16:20:09.361822 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:09.361785 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" event={"ID":"3d0efab9-17c2-4f08-9433-02670766f69c","Type":"ContainerStarted","Data":"5fb94406767561ed1759940da4ffc3770e72c0b3fb8961d21e45b3edecb5164f"} Oct 30 16:20:09.363954 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:09.363931 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9fbq" event={"ID":"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b","Type":"ContainerStarted","Data":"3d381ef66e3a1451146fafdac681969ff079553176f1c75731ded5cd083576d2"} Oct 30 16:20:09.364099 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:09.364001 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9fbq" event={"ID":"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b","Type":"ContainerStarted","Data":"eb43177818a29fa9d149ed0f29291e844551e4729e5a8e096d44c625ff81369c"} Oct 30 16:20:09.365989 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:09.365946 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" event={"ID":"be9a1b81-45d0-46a0-a771-84d88046f28e","Type":"ContainerStarted","Data":"ad6ae44132beb983d83ed2ae253566ddad2ce6feb3b3a4b5102d9b2219e152a4"} Oct 30 16:20:09.366102 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:09.366012 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" event={"ID":"be9a1b81-45d0-46a0-a771-84d88046f28e","Type":"ContainerStarted","Data":"2ac827c754388327111b693874bae4aa6a83d10e1fb347e3adcd87435963f267"} Oct 30 16:20:09.366102 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:09.366031 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" event={"ID":"be9a1b81-45d0-46a0-a771-84d88046f28e","Type":"ContainerStarted","Data":"9d139c00e89cd2bad4ee5e2c85a02edb285ffe9c68120483eb2ff5cbff7c8e6c"} Oct 30 16:20:09.408498 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:09.408450 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" podStartSLOduration=2.9536088080000003 podStartE2EDuration="4.408436805s" totalImagesPullingTime="1.454827997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:09.408080003 +0000 UTC m=+41.965637556" watchObservedRunningTime="2026-10-30 16:20:09.408436805 +0000 UTC m=+41.965994327" Oct 30 16:20:09.461800 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:09.461750 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b9fbq" podStartSLOduration=2.989833415 podStartE2EDuration="4.461737144s" totalImagesPullingTime="1.371424661s" totalInitContainerRuntime="100.479068ms" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:09.46095193 +0000 UTC m=+42.018509455" watchObservedRunningTime="2026-10-30 16:20:09.461737144 +0000 UTC m=+42.019294668" Oct 30 16:20:09.509765 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:09.509710 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" podStartSLOduration=2.009796607 podStartE2EDuration="3.509700243s" totalImagesPullingTime="1.499903636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:09.508726249 +0000 UTC m=+42.066283774" watchObservedRunningTime="2026-10-30 16:20:09.509700243 +0000 UTC m=+42.067257758" Oct 30 16:20:10.482237 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.482209 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5d79986586-bccmm"] Oct 30 16:20:10.484880 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.484865 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.503077 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.503057 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 16:20:10.517011 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.516984 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 16:20:10.517144 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.517082 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 16:20:10.517222 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.517204 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-g5m2c\"" Oct 30 16:20:10.517270 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.517225 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-85pv7n2is4r7c\"" Oct 30 16:20:10.517270 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.517208 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 16:20:10.532355 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.532331 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d79986586-bccmm"] Oct 30 16:20:10.576878 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.576847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-client-ca-bundle\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.577029 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.576901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5beb53b1-b44b-47eb-84d8-31660627d4a3-metrics-server-audit-profiles\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.577029 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.577007 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5beb53b1-b44b-47eb-84d8-31660627d4a3-audit-log\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.577101 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.577056 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-secret-metrics-server-tls\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.577101 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.577089 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpkkc\" (UniqueName: \"kubernetes.io/projected/5beb53b1-b44b-47eb-84d8-31660627d4a3-kube-api-access-xpkkc\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.577179 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.577161 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-secret-metrics-server-client-certs\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.577215 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.577196 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5beb53b1-b44b-47eb-84d8-31660627d4a3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.678279 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.678246 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5beb53b1-b44b-47eb-84d8-31660627d4a3-audit-log\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.678438 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.678287 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-secret-metrics-server-tls\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.678438 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.678331 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpkkc\" (UniqueName: \"kubernetes.io/projected/5beb53b1-b44b-47eb-84d8-31660627d4a3-kube-api-access-xpkkc\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.678438 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.678383 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-secret-metrics-server-client-certs\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.678438 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.678421 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5beb53b1-b44b-47eb-84d8-31660627d4a3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.678659 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.678464 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-client-ca-bundle\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.678659 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.678517 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5beb53b1-b44b-47eb-84d8-31660627d4a3-metrics-server-audit-profiles\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.678751 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.678728 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5beb53b1-b44b-47eb-84d8-31660627d4a3-audit-log\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.679217 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.679189 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5beb53b1-b44b-47eb-84d8-31660627d4a3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.679546 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.679521 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5beb53b1-b44b-47eb-84d8-31660627d4a3-metrics-server-audit-profiles\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.682187 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.682164 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-secret-metrics-server-tls\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.682290 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.682210 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-client-ca-bundle\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.682290 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.682227 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-secret-metrics-server-client-certs\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.703913 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.703887 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpkkc\" (UniqueName: \"kubernetes.io/projected/5beb53b1-b44b-47eb-84d8-31660627d4a3-kube-api-access-xpkkc\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:10.795887 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:10.795853 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:11.109626 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:11.109395 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d79986586-bccmm"] Oct 30 16:20:11.113035 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:20:11.113006 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5beb53b1_b44b_47eb_84d8_31660627d4a3.slice/crio-9a65ebc67454e0ccf639133a0a0dcb08ca5b36c47b5434e35bbae75bed1115da WatchSource:0}: Error finding container 9a65ebc67454e0ccf639133a0a0dcb08ca5b36c47b5434e35bbae75bed1115da: Status 404 returned error can't find the container with id 9a65ebc67454e0ccf639133a0a0dcb08ca5b36c47b5434e35bbae75bed1115da Oct 30 16:20:11.374382 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:11.374303 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d79986586-bccmm" event={"ID":"5beb53b1-b44b-47eb-84d8-31660627d4a3","Type":"ContainerStarted","Data":"9a65ebc67454e0ccf639133a0a0dcb08ca5b36c47b5434e35bbae75bed1115da"} Oct 30 16:20:12.378645 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:12.378615 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d79986586-bccmm" event={"ID":"5beb53b1-b44b-47eb-84d8-31660627d4a3","Type":"ContainerStarted","Data":"54b9bd070d77f4a824cb5b639bf8cce6af08f6eda04b4d72ebe33220476c336e"} Oct 30 16:20:12.425493 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:12.425443 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5d79986586-bccmm" podStartSLOduration=1.231338955 podStartE2EDuration="2.425427263s" totalImagesPullingTime="1.194088308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:12.42454125 +0000 UTC m=+44.982098775" watchObservedRunningTime="2026-10-30 16:20:12.425427263 +0000 UTC m=+44.982984788" Oct 30 16:20:18.369057 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:18.369023 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:22.310725 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:22.310698 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:30.796189 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:30.796154 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:30.796682 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:30.796221 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:32.760024 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.759993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:20:32.760432 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.760038 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:32.790249 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.790223 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:20:32.790383 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.790254 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:20:32.803390 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.803361 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:32.803497 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.803401 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:20:32.860781 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.860752 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skbh\" (UniqueName: \"kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh\") pod \"network-check-target-mv4lc\" (UID: \"d90a91c8-f7a6-4526-a13c-ce484713b6b5\") " pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:20:32.876776 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.876754 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:20:32.897634 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.897610 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:20:32.904512 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.904490 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skbh\" (UniqueName: \"kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh\") pod \"network-check-target-mv4lc\" (UID: \"d90a91c8-f7a6-4526-a13c-ce484713b6b5\") " pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:20:32.923074 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.923053 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2ph4\"" Oct 30 16:20:32.928149 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.928131 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:20:32.937463 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.937444 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bhx7q\"" Oct 30 16:20:32.942704 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.942681 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:32.955498 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.955468 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t25rz\"" Oct 30 16:20:32.960617 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:32.960581 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:20:33.228621 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:33.228588 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4856h"] Oct 30 16:20:33.241777 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:33.241749 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lt2pz"] Oct 30 16:20:33.244044 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:20:33.244021 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36b8f12_981e_46a8_b422_632babc5d4a7.slice/crio-5e088f6ab1c408134be111533036761823177e97a6e9b9493acac296f80f53f2 WatchSource:0}: Error finding container 5e088f6ab1c408134be111533036761823177e97a6e9b9493acac296f80f53f2: Status 404 returned error can't find the container with id 5e088f6ab1c408134be111533036761823177e97a6e9b9493acac296f80f53f2 Oct 30 16:20:33.277028 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:33.276888 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mv4lc"] Oct 30 16:20:33.279344 ip-10-0-137-219 kubenswrapper[2583]: W1030 16:20:33.279318 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90a91c8_f7a6_4526_a13c_ce484713b6b5.slice/crio-066e878b8059c156ab32b4d15b8589b9e29037a987260ef92e01207dd3249417 WatchSource:0}: Error finding container 066e878b8059c156ab32b4d15b8589b9e29037a987260ef92e01207dd3249417: Status 404 returned error can't find the container with id 066e878b8059c156ab32b4d15b8589b9e29037a987260ef92e01207dd3249417 Oct 30 16:20:33.449037 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:33.448998 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mv4lc" event={"ID":"d90a91c8-f7a6-4526-a13c-ce484713b6b5","Type":"ContainerStarted","Data":"066e878b8059c156ab32b4d15b8589b9e29037a987260ef92e01207dd3249417"} Oct 30 16:20:33.450191 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:33.450168 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4856h" event={"ID":"ddebdc11-43b3-4787-a3ff-bf95150f8d27","Type":"ContainerStarted","Data":"8dafd2b2cd7e4dfa578a58bf89762fedea6e1d7345757f3360da3e5bceb9e3f8"} Oct 30 16:20:33.451289 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:33.451264 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lt2pz" event={"ID":"b36b8f12-981e-46a8-b422-632babc5d4a7","Type":"ContainerStarted","Data":"5e088f6ab1c408134be111533036761823177e97a6e9b9493acac296f80f53f2"} Oct 30 16:20:33.728004 ip-10-0-137-219 kubenswrapper[2583]: E1030 16:20:33.727914 2583 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:20:34.457703 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:34.457546 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4856h" event={"ID":"ddebdc11-43b3-4787-a3ff-bf95150f8d27","Type":"ContainerStarted","Data":"c6d1218450187994789d501478ed66b547a60e3fc15d5ce9b1d131149b4b7c2b"} Oct 30 16:20:34.457703 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:34.457657 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4856h" event={"ID":"ddebdc11-43b3-4787-a3ff-bf95150f8d27","Type":"ContainerStarted","Data":"783d2ba581880797682de340300ae73c1fbffafc67da537c2a7bdf2802018c1c"} Oct 30 16:20:34.505385 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:34.505322 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4856h" podStartSLOduration=65.547107593 podStartE2EDuration="1m6.505303212s" totalImagesPullingTime="958.195619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:34.504795521 +0000 UTC m=+67.062353045" watchObservedRunningTime="2026-10-30 16:20:34.505303212 +0000 UTC m=+67.062860739" Oct 30 16:20:37.469743 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:37.469704 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mv4lc" event={"ID":"d90a91c8-f7a6-4526-a13c-ce484713b6b5","Type":"ContainerStarted","Data":"b5553a89c51a7c1aa944190a2822ea07d9b17aaddba3f5cb0eeca70549f36588"} Oct 30 16:20:37.470242 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:37.469848 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:20:37.515490 ip-10-0-137-219 kubenswrapper[2583]: I1030 16:20:37.515429 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mv4lc" podStartSLOduration=66.271474984 podStartE2EDuration="1m9.515409053s" totalImagesPullingTime="3.243934069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:37.513508462 +0000 UTC m=+70.071065988" watchObservedRunningTime="2026-10-30 16:20:37.515409053 +0000 UTC m=+70.072966582" Oct 30 16:20:38.185576 ip-10-0-137-219 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 16:20:38.201419 ip-10-0-137-219 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 16:20:38.201647 ip-10-0-137-219 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 16:20:38.201867 ip-10-0-137-219 systemd[1]: kubelet.service: Consumed 3.827s CPU time. Oct 30 16:20:38.211669 ip-10-0-137-219 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:20:38.311412 ip-10-0-137-219 kubenswrapper[5838]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:20:38.311412 ip-10-0-137-219 kubenswrapper[5838]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:20:38.311412 ip-10-0-137-219 kubenswrapper[5838]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:20:38.311412 ip-10-0-137-219 kubenswrapper[5838]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:20:38.313464 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313375 5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:38.313464 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313463 5838 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:38.313464 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313468 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313471 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313474 5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313478 5838 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313480 5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313483 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313486 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313488 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313497 5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313500 5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313503 5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313505 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313508 5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313511 5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313513 5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313516 5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313519 5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313521 5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313524 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313527 5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:38.313569 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313529 5838 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313532 5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313534 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313537 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313540 5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313542 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313545 5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313548 5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313551 5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313553 5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313556 5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313559 5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313562 5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313565 5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313568 5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313571 5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313580 5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313583 5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313586 5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:38.314122 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313589 5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313592 5838 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313594 5838 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313597 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313601 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313603 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313606 5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313609 5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313611 5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313614 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313616 5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313619 5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313621 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313624 5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313627 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313629 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313632 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313639 5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313642 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313645 5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:38.314604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313647 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313650 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313653 5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313655 5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313659 5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313663 5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313669 5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313672 5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313675 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313678 5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313685 5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313688 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313690 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313693 5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313696 5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313698 5838 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313701 5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313703 5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313706 5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:38.315166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313708 5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313711 5838 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313714 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313718 5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313722 5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313724 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313728 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313731 5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313734 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313737 5838 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313741 5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313743 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313746 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313749 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313751 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313754 5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313757 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313760 5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313763 5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:38.315640 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313766 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313770 5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313773 5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313775 5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313778 5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313781 5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.313784 5838 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314600 5838 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314609 5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314613 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314617 5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314620 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314623 5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314625 5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314628 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314631 5838 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314633 5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314636 5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314639 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:38.316166 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314641 5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314644 5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314647 5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314650 5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314654 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314656 5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314659 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314661 5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314664 5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314666 5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314669 5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314672 5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314674 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314677 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314680 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314682 5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314685 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314687 5838 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314690 5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:38.316658 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314693 5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314695 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314698 5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314700 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314703 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314705 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314710 5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314713 5838 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314716 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314719 5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314722 5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314724 5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314727 5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314730 5838 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314732 5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314735 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314737 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314740 5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314743 5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:38.317151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314746 5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314748 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314751 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314754 5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314756 5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314759 5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314762 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314764 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314767 5838 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314770 5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314772 5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314775 5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314778 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314781 5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314784 5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314787 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314789 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314792 5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314795 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:38.317622 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314797 5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314800 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314802 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314805 5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314807 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314810 5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314813 5838 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314816 5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314818 5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314821 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314824 5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314827 5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314829 5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314843 5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314845 5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314848 5838 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314851 5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314853 5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314856 5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314859 5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:38.318116 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314861 5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314864 5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314866 5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314870 5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314872 5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314875 5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314877 5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314880 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314883 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314885 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314888 5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314890 5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314893 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314895 5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314900 5838 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314903 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.314905 5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315424 5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315429 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315432 5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:38.318604 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315435 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315438 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315441 5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315444 5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315446 5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315449 5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315451 5838 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315454 5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315457 5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315459 5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315462 5838 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315465 5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315467 5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315470 5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315473 5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315475 5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315478 5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315481 5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315483 5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315487 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:38.319113 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315490 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315493 5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315496 5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315498 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315501 5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315503 5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315508 5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315511 5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315513 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315516 5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315519 5838 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315523 5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315527 5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315530 5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315532 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315535 5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315538 5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315540 5838 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315543 5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:38.319849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315545 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315548 5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315551 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315553 5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315557 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315560 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315562 5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315565 5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315568 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315570 5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315573 5838 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315575 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315578 5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315580 5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315583 5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315586 5838 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315588 5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315591 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315593 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315597 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:38.320700 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315600 5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315603 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315605 5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315608 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315611 5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315614 5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315616 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315619 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315622 5838 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315625 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315627 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315630 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315632 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315635 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315637 5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315640 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315643 5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315645 5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315654 5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:38.321354 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315659 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315661 5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315681 5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315686 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315689 5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315693 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315696 5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315700 5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315704 5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315708 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315712 5838 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315714 5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315719 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315722 5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315724 5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315728 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315730 5838 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315733 5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315736 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:38.321978 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315739 5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315741 5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315744 5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315747 5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315749 5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.315752 5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315843 5838 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315855 5838 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315862 5838 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315867 5838 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315871 5838 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315874 5838 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315879 5838 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315883 5838 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315886 5838 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315889 5838 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315893 5838 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315896 5838 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315899 5838 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315902 5838 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315905 5838 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315908 5838 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315912 5838 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:20:38.322465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315915 5838 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315920 5838 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315924 5838 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315928 5838 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315931 5838 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315936 5838 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315940 5838 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315943 5838 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315946 5838 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315950 5838 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315953 5838 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315956 5838 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315959 5838 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315962 5838 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315965 5838 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315986 5838 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315989 5838 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315992 5838 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315995 5838 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.315999 5838 flags.go:64] FLAG: --enable-server="true" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316002 5838 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316007 5838 flags.go:64] FLAG: --event-burst="100" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316010 5838 flags.go:64] FLAG: --event-qps="50" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316013 5838 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:20:38.323018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316016 5838 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316020 5838 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316025 5838 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316028 5838 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316031 5838 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316034 5838 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316038 5838 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316041 5838 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316045 5838 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316048 5838 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316052 5838 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316055 5838 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316058 5838 flags.go:64] FLAG: --feature-gates="" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316064 5838 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316067 5838 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316071 5838 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316074 5838 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316077 5838 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316080 5838 flags.go:64] FLAG: --help="false" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316083 5838 flags.go:64] FLAG: --hostname-override="ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316086 5838 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316089 5838 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316092 5838 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316096 5838 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:20:38.323611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316099 5838 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316102 5838 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316105 5838 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316108 5838 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316111 5838 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316114 5838 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316117 5838 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316120 5838 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316123 5838 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316127 5838 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316130 5838 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316133 5838 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316136 5838 flags.go:64] FLAG: --lock-file="" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316139 5838 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316142 5838 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316146 5838 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316151 5838 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316155 5838 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316159 5838 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316163 5838 flags.go:64] FLAG: --logging-format="text" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316175 5838 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316180 5838 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316183 5838 flags.go:64] FLAG: --manifest-url="" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316186 5838 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316191 5838 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:20:38.324226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316194 5838 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316198 5838 flags.go:64] FLAG: --max-pods="110" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316201 5838 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316205 5838 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316208 5838 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316211 5838 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316214 5838 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316218 5838 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316221 5838 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316227 5838 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316231 5838 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316234 5838 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316237 5838 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316240 5838 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316243 5838 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316246 5838 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316249 5838 flags.go:64] FLAG: --port="10250" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316252 5838 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316256 5838 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e7ad6bbabb8f54d9" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316259 5838 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316262 5838 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316265 5838 flags.go:64] FLAG: --register-node="true" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316268 5838 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316272 5838 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316275 5838 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316279 5838 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:20:38.324855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316283 5838 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316287 5838 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316291 5838 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316294 5838 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316297 5838 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316300 5838 flags.go:64] FLAG: --runonce="false" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316304 5838 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316307 5838 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316310 5838 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316313 5838 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316316 5838 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316319 5838 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316323 5838 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316326 5838 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316329 5838 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316332 5838 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316335 5838 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316338 5838 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316341 5838 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316344 5838 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316347 5838 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316352 5838 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316355 5838 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316358 5838 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316366 5838 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:20:38.325599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316369 5838 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316372 5838 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316375 5838 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316378 5838 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316381 5838 flags.go:64] FLAG: --v="2" Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316386 5838 flags.go:64] FLAG: --version="false" Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316391 5838 flags.go:64] FLAG: --vmodule="" Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316396 5838 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316399 5838 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316534 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316538 5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316542 5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316545 5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316548 5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316551 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316554 5838 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316557 5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316559 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316562 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316564 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316567 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:38.326268 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316570 5838 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316573 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316575 5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316578 5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316582 5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316586 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316589 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316592 5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316595 5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316597 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316600 5838 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316603 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316605 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316608 5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316610 5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316613 5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316616 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316620 5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316622 5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:38.326791 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316625 5838 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316629 5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316632 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316634 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316637 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316640 5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316642 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316645 5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316648 5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316650 5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316653 5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316655 5838 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316659 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316661 5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316664 5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316667 5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316670 5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316673 5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316675 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316678 5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:38.327507 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316680 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316683 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316685 5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316688 5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316691 5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316693 5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316696 5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316698 5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316701 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316704 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316708 5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316710 5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316713 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316721 5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316724 5838 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316727 5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316729 5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316732 5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316734 5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316737 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:38.328167 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316739 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316742 5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316745 5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316747 5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316750 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316753 5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316755 5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316758 5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316760 5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316763 5838 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316765 5838 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316768 5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316771 5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316773 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316776 5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316778 5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316781 5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316785 5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316788 5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:38.328742 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316791 5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316795 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316798 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316802 5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316805 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316807 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316811 5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316814 5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316817 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316820 5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316822 5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316825 5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316828 5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316831 5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316833 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316836 5838 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:38.329499 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.316861 5838 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316900 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316905 5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316908 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316911 5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316914 5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316917 5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316919 5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316922 5838 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316925 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316928 5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316931 5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316933 5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316936 5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316939 5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316941 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316944 5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316946 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316949 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316951 5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:38.330400 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316955 5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316958 5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316963 5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316966 5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316983 5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316986 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316989 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316992 5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316995 5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.316997 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317000 5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317003 5838 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317005 5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317008 5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317011 5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317013 5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317016 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317019 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317021 5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317024 5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:38.331184 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317027 5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317029 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317032 5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317035 5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317037 5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317040 5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317042 5838 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317045 5838 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317048 5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317050 5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317053 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317055 5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317061 5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317064 5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317069 5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317073 5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317076 5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317080 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317083 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:38.331858 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317086 5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317088 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317091 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317093 5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317096 5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317099 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317101 5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317104 5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317107 5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317111 5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317114 5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317116 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317119 5838 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317122 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317124 5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317127 5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317130 5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317133 5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317135 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317138 5838 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:38.332374 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317141 5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317143 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317146 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317148 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317151 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317156 5838 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317159 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317164 5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317167 5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317171 5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317175 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317178 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317181 5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317184 5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317188 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317190 5838 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317193 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317196 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317198 5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:38.332864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317201 5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317203 5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317206 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317209 5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317211 5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317214 5838 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317216 5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317219 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.317222 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.322755 5838 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.322773 5838 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.322793 5838 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.322798 5838 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322877 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322883 5838 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322887 5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322891 5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322895 5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322899 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:38.333351 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322901 5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322904 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322907 5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322910 5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322913 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322916 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322919 5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322921 5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322924 5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322926 5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322929 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322932 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322934 5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322937 5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322939 5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322942 5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322945 5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322948 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322950 5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:38.333826 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322953 5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322961 5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322964 5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322979 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322986 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322989 5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322992 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322994 5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.322997 5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323000 5838 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323003 5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323005 5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323008 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323011 5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323014 5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323017 5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323019 5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323022 5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323025 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:38.334316 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323028 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323030 5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323033 5838 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323037 5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323039 5838 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323042 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323045 5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323047 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323051 5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323055 5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323058 5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323061 5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323064 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323067 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323070 5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323073 5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323076 5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323079 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323082 5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:38.334822 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323085 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323087 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323090 5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323093 5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323095 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323098 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323100 5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323103 5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323106 5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323108 5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323111 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323113 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323116 5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323119 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323121 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323124 5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323128 5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323131 5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323133 5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323136 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:38.335443 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323138 5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323141 5838 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323144 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323147 5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323149 5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323152 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323155 5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323157 5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323160 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323162 5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323165 5838 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323168 5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323171 5838 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323173 5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323176 5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323178 5838 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323181 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323183 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323186 5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323188 5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:38.336040 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323192 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323194 5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323197 5838 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.323226 5838 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323368 5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323372 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323375 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323379 5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323381 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323385 5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323387 5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323391 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323393 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323396 5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323399 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323402 5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:38.336830 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323404 5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323407 5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323410 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323413 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323415 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323418 5838 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323421 5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323423 5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323426 5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323428 5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323431 5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323434 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323436 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323439 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323441 5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323444 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323446 5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323449 5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323452 5838 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323454 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:38.337463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323457 5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323460 5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323462 5838 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323465 5838 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323467 5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323470 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323473 5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323476 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323478 5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323481 5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323483 5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323486 5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323488 5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323491 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323493 5838 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323496 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323499 5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323501 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323504 5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323506 5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:38.338039 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323509 5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323513 5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323517 5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323521 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323524 5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323526 5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323529 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323532 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323534 5838 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323537 5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323539 5838 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323542 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323544 5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323547 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323549 5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323552 5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323555 5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323557 5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323560 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:38.338528 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323563 5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323566 5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323569 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323571 5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323574 5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323576 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323579 5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323582 5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323584 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323587 5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323589 5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323592 5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323594 5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323597 5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323599 5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323602 5838 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323604 5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323607 5838 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323609 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:38.339009 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323612 5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323615 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323618 5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323621 5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323625 5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323629 5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323632 5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323635 5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323638 5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323640 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323642 5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323645 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323655 5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323660 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323664 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:38.339463 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323667 5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.323691 5838 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323741 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323746 5838 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323750 5838 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323753 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323755 5838 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323758 5838 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323760 5838 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323763 5838 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323766 5838 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323768 5838 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323771 5838 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323774 5838 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323776 5838 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:38.349226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323779 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323782 5838 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323785 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323787 5838 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323790 5838 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323794 5838 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323798 5838 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323801 5838 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323804 5838 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323807 5838 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323810 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323812 5838 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323815 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323818 5838 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323820 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323823 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323826 5838 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323829 5838 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323831 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:38.349601 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323834 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323837 5838 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323839 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323842 5838 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323845 5838 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323848 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323850 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323853 5838 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323855 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323859 5838 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323861 5838 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323864 5838 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323866 5838 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323869 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323872 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323875 5838 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323877 5838 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323880 5838 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323882 5838 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:38.350159 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323885 5838 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323888 5838 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323890 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323893 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323898 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323901 5838 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323903 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323906 5838 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323908 5838 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323911 5838 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323914 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323916 5838 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323919 5838 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323922 5838 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323925 5838 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323927 5838 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323930 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323932 5838 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323935 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323937 5838 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:38.351057 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323940 5838 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:38.350871 ip-10-0-137-219 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323942 5838 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323945 5838 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323950 5838 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323953 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323955 5838 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323958 5838 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323961 5838 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323963 5838 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323978 5838 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323983 5838 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323986 5838 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323990 5838 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323995 5838 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.323999 5838 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324002 5838 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324006 5838 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324009 5838 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324012 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:38.352226 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324015 5838 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324018 5838 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324021 5838 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324024 5838 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324026 5838 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324029 5838 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324032 5838 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324035 5838 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324038 5838 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324040 5838 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324043 5838 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324047 5838 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324049 5838 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324052 5838 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324055 5838 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:38.324057 5838 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.324125 5838 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.325130 5838 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.325192 5838 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 16:20:38.353136 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.325373 5838 server.go:1014] "Starting client certificate rotation" Oct 30 16:20:38.353928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.325519 5838 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:20:38.353928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.325592 5838 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:14:27 +0000 UTC" deadline="2028-06-06 08:51:27.952157727 +0000 UTC" Oct 30 16:20:38.353928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.325629 5838 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14032h30m49.626530961s" Oct 30 16:20:38.353928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.325810 5838 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:20:38.353928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.326999 5838 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:20:38.353928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.328670 5838 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:20:38.353928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.331044 5838 log.go:25] "Validated CRI v1 image API" Oct 30 16:20:38.353928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.331821 5838 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:20:38.353928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.333806 5838 fs.go:135] Filesystem UUIDs: map[379128d4-6b47-4be8-a1db-fbeb01481d55:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c1556daf-498c-4e61-a4c5-127f44fbd06b:/dev/nvme0n1p3] Oct 30 16:20:38.354391 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.333823 5838 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/066e878b8059c156ab32b4d15b8589b9e29037a987260ef92e01207dd3249417/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/066e878b8059c156ab32b4d15b8589b9e29037a987260ef92e01207dd3249417/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fcc8b41f6f78812d8fd579ca43e70a488b6f9659ef2efa107fdef1be43b3c4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fcc8b41f6f78812d8fd579ca43e70a488b6f9659ef2efa107fdef1be43b3c4d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13fe18553811e4ef9d4d2cd07996a2abb66728c7d29d2f1a67381e0009c5d1f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13fe18553811e4ef9d4d2cd07996a2abb66728c7d29d2f1a67381e0009c5d1f6/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1796ff68883d583b3245e9a58383981660e62e063b39996985bf98ea72810169/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1796ff68883d583b3245e9a58383981660e62e063b39996985bf98ea72810169/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22376cc752f611717ad034681e2aeb5629a1f1848734d9520754118aea8e962a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22376cc752f611717ad034681e2aeb5629a1f1848734d9520754118aea8e962a/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27fd84afe85bce8c2321e1b1b85b813f4c9ac089819b3c63c22e4e61bf8aea4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27fd84afe85bce8c2321e1b1b85b813f4c9ac089819b3c63c22e4e61bf8aea4a/userdata/shm major:0 minor:257 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e8b300bb7ba02af7d2ec639960543489b56a6fe8f316660cfca5d4edf70e072/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e8b300bb7ba02af7d2ec639960543489b56a6fe8f316660cfca5d4edf70e072/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37d8bfdd628d25d8061da4f3a87c1a8b41de644e58a636ec0551ec9cb59ac51f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37d8bfdd628d25d8061da4f3a87c1a8b41de644e58a636ec0551ec9cb59ac51f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d47927b38cdc342416f26d788c9a3569434173c31a5a4746c2ee8b7bdd30fdd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d47927b38cdc342416f26d788c9a3569434173c31a5a4746c2ee8b7bdd30fdd/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5aeb3fbd398c51d8dc64868da92878d26a596a7beb445896d953f1650f7cc0c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5aeb3fbd398c51d8dc64868da92878d26a596a7beb445896d953f1650f7cc0c1/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e088f6ab1c408134be111533036761823177e97a6e9b9493acac296f80f53f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e088f6ab1c408134be111533036761823177e97a6e9b9493acac296f80f53f2/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62e319000ca88b93d246d429f68a234857e121cbee4eb5504346e3d0ba39c0b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62e319000ca88b93d246d429f68a234857e121cbee4eb5504346e3d0ba39c0b8/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dafd2b2cd7e4dfa578a58bf89762fedea6e1d7345757f3360da3e5bceb9e3f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dafd2b2cd7e4dfa578a58bf89762fedea6e1d7345757f3360da3e5bceb9e3f8/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/948f3d754808f828efa22bc27a3d7c31b701d85424db6ceb405de474567aaa84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/948f3d754808f828efa22bc27a3d7c31b701d85424db6ceb405de474567aaa84/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a65ebc67454e0ccf639133a0a0dcb08ca5b36c47b5434e35bbae75bed1115da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a65ebc67454e0ccf639133a0a0dcb08ca5b36c47b5434e35bbae75bed1115da/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6af2dcb8be6646a151b1dcbf23d69433191f530b283c91882258c5aa90bbd5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6af2dcb8be6646a151b1dcbf23d69433191f530b283c91882258c5aa90bbd5b/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b75f55b1eabe52dfe9a1d14564c828142c3dae665cf2a30ff0180f333b3d3ab7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b75f55b1eabe52dfe9a1d14564c828142c3dae665cf2a30ff0180f333b3d3ab7/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2228af531f82f28dc3beb9a5bb32696a044e1a6f85a29a9a873997054ee9940/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2228af531f82f28dc3beb9a5bb32696a044e1a6f85a29a9a873997054ee9940/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db4b67644dc5923ad73e9ba0e1a4aa59668646dba0c302557567627c4838b297/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db4b67644dc5923ad73e9ba0e1a4aa59668646dba0c302557567627c4838b297/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6b17f375c7f7270b0f267083307e091303bd28d80da37550d26258867925c02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6b17f375c7f7270b0f267083307e091303bd28d80da37550d26258867925c02/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/236ec382-dbd5-4974-9454-43b39e990c77/volumes/kubernetes.io~projected/kube-api-access-ln9pm:{mountpoint:/var/lib/kubelet/pods/236ec382-dbd5-4974-9454-43b39e990c77/volumes/kubernetes.io~projected/kube-api-access-ln9pm major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/236ec382-dbd5-4974-9454-43b39e990c77/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/236ec382-dbd5-4974-9454-43b39e990c77/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30634e3d-5f01-4b9b-88ad-03893a2b444e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/30634e3d-5f01-4b9b-88ad-03893a2b444e/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d0efab9-17c2-4f08-9433-02670766f69c/volumes/kubernetes.io~projected/kube-api-access-4n7bv:{mountpoint:/var/lib/kubelet/pods/3d0efab9-17c2-4f08-9433-02670766f69c/volumes/kubernetes.io~projected/kube-api-access-4n7bv major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d0efab9-17c2-4f08-9433-02670766f69c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3d0efab9-17c2-4f08-9433-02670766f69c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d0efab9-17c2-4f08-9433-02670766f69c/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/3d0efab9-17c2-4f08-9433-02670766f69c/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e07b376-b6a8-4d05-b6f4-705695c0e92b/volumes/kubernetes.io~projected/kube-api-access-2jd74:{mountpoint:/var/lib/kubelet/pods/3e07b376-b6a8-4d05-b6f4-705695c0e92b/volumes/kubernetes.io~projected/kube-api-access-2jd74 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b13dfe3-9da5-4be8-9051-b1bbff1b083b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4b13dfe3-9da5-4be8-9051-b1bbff1b083b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b13dfe3-9da5-4be8-9051-b1bbff1b083b/volumes/kubernetes.io~projected/kube-api-access-2rbfc:{mountpoint:/var/lib/kubelet/pods/4b13dfe3-9da5-4be8-9051-b1bbff1b083b/volumes/kubernetes.io~projected/kube-api-access-2rbfc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b13dfe3-9da5-4be8-9051-b1bbff1b083b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4b13dfe3-9da5-4be8-9051-b1bbff1b083b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2/volumes/kubernetes.io~projected/kube-api-access-qkz5f:{mountpoint:/var/lib/kubelet/pods/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2/volumes/kubernetes.io~projected/kube-api-access-qkz5f major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5beb53b1-b44b-47eb-84d8-31660627d4a3/volumes/kubernetes.io~projected/kube-api-access-xpkkc:{mountpoint:/var/lib/kubelet/pods/5beb53b1-b44b-47eb-84d8-31660627d4a3/volumes/kubernetes.io~projected/kube-api-access-xpkkc major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5beb53b1-b44b-47eb-84d8-31660627d4a3/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/5beb53b1-b44b-47eb-84d8-31660627d4a3/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5beb53b1-b44b-47eb-84d8-31660627d4a3/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/5beb53b1-b44b-47eb-84d8-31660627d4a3/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5beb53b1-b44b-47eb-84d8-31660627d4a3/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/5beb53b1-b44b-47eb-84d8-31660627d4a3/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67036dae-0e4a-473b-af97-e0ee20d824b8/volumes/kubernetes.io~projected/kube-api-access-57d5q:{mountpoint:/var/lib/kubelet/pods/67036dae-0e4a-473b-af97-e0ee20d824b8/volumes/kubernetes.io~projected/kube-api-access-57d5q major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/791cb17f-1c21-414c-9f90-196b26aa3570/volumes/kubernetes.io~projected/kube-api-access-p8zmd:{mountpoint:/var/lib/kubelet/pods/791cb17f-1c21-414c-9f90-196b26aa3570/volumes/kubernetes.io~projected/kube-api-access-p8zmd major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/791cb17f-1c21-414c-9f90-196b26aa3570/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/791cb17f-1c21-414c-9f90-196b26aa3570/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81a6d8ed-431a-4b9b-bd73-d13b01208249/volumes/kubernetes.io~projected/kube-api-access-bk6g4:{mountpoint:/var/lib/kubelet/pods/81a6d8ed-431a-4b9b-bd73-d13b01208249/volumes/kubernetes.io~projected/kube-api-access-bk6g4 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ca822f3-98ef-487f-a89d-1a1004beeda1/volumes/kubernetes.io~projected/kube-api-access-fdx66:{mountpoint:/var/lib/kubelet/pods/9ca822f3-98ef-487f-a89d-1a1004beeda1/volumes/kubernetes.io~projected/kube-api-access-fdx66 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ca822f3-98ef-487f-a89d-1a1004beeda1/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/9ca822f3-98ef-487f-a89d-1a1004beeda1/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2196112-4441-49a3-b48f-1b80bf426e29/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a2196112-4441-49a3-b48f-1b80bf426e29/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2196112-4441-49a3-b48f-1b80bf426e29/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a2196112-4441-49a3-b48f-1b80bf426e29/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2196112-4441-49a3-b48f-1b80bf426e29/volumes/kubernetes.io~projected/kube-api-access-4qnf9:{mountpoint:/var/lib/kubelet/pods/a2196112-4441-49a3-b48f-1b80bf426e29/volumes/kubernetes.io~projected/kube-api-access-4qnf9 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b/volumes/kubernetes.io~projected/kube-api-access-lwhlq:{mountpoint:/var/lib/kubelet/pods/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b/volumes/kubernetes.io~projected/kube-api-access-lwhlq major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b36b8f12-981e-46a8-b422-632babc5d4a7/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b36b8f12-981e-46a8-b422-632babc5d4a7/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be9a1b81-45d0-46a0-a771-84d88046f28e/volumes/kubernetes.io~projected/kube-api-access-6tn95:{mountpoint:/var/lib/kubelet/pods/be9a1b81-45d0-46a0-a771-84d88046f28e/volumes/kubernetes.io~projected/kube-api-access-6tn95 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be9a1b81-45d0-46a0-a771-84d88046f28e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/be9a1b81-45d0-46a0-a771-84d88046f28e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be9a1b81-45d0-46a0-a771-84d88046f28e/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/be9a1b81-45d0-46a0-a771-84d88046f28e/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd2b0c8e-8194-456d-8299-c43b1688404a/volumes/kubernetes.io~projected/kube-api-access-cblkc:{mountpoint:/var/lib/kubelet/pods/cd2b0c8e-8194-456d-8299-c43b1688404a/volumes/kubernetes.io~projected/kube-api-access-cblkc major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d90a91c8-f7a6-4526-a13c-ce484713b6b5/volumes/kubernetes.io~projected/kube-api-access-8skbh:{mountpoint:/var/lib/kubelet/pods/d90a91c8-f7a6-4526-a13c-ce484713b6b5/volumes/kubernetes.io~projected/kube-api-access-8skbh major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddebdc11-43b3-4787-a3ff-bf95150f8d27/volumes/kubernetes.io~projected/kube-api-access-s6c2l:{mountpoint:/var/lib/kubelet/pods/ddebdc11-43b3-4787-a3ff-bf95150f8d27/volumes/kubernetes.io~projected/kube-api-access-s6c2l major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddebdc11-43b3-4787-a3ff-bf95150f8d27/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/ddebdc11-43b3-4787-a3ff-bf95150f8d27/volumes/kubernetes.io~secret/metrics-certs major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb2c72e6-372c-41c1-ba29-d47c0b521f34/volumes/kubernetes.io~projected/kube-api-access-99d2h:{mountpoint:/var/lib/kubelet/pods/fb2c72e6-372c-41c1-ba29-d47c0b521f34/volumes/kubernetes.io~projected/kube-api-access-99d2h major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/06942e71e57c209900b5669c053ceaedb644ceed159974b50e4a406bcb956610/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e3609bbc675eb77fa4248d8be63b267b3a50d2d6754d693978e93c6937ee184c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/dc55b71e715e4e546f5787928d8525360479703ef5a14311568fff21e9b47043/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/1186f988d76275358afd4e689c9ac21aa35df9573f18499309d52401d61dc90d/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/592810f59b697cee3fde70e124c31f97bb445ba65223b9c9fc705ab5603f5f02/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c17b05a8d903927617b666cb14778c411ab45359377a2a5b1e3386dd0b5acc87/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8d65cf32f470495611c55a537fdca82dfcfc559f11eae9c45ef27bf6cb834d55/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/befb81da621678fa4062aa3ec3f1c9e23ddc7ac3dd280b9e7271258983f249e4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c5ad877fb12f9059fb3600fa7d3e38a4a5e375e318b01b2ea3203b6194cf167a/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/da7a539e4a890358e649484cbfbbf5547b6a741fe94f860a9aec373f57d06f8a/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/3892afbc73a1eb3d2735265f8627b17a2a22bf4f0de7487c2ffe234ea26eb76f/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/3ee12ed00b81b013a6491dcd9c05793775d127f76d84e5822ca10c43ca50f754/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-151:{mountpoint:/var/lib/containers/storage/overlay/710ebe84822ecd0fd0cafca0b1a35b6fa935818f59ec9f5b6f204e2863c63b65/merged major:0 minor:151 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ed2df9153ef00950712ad4a24a010aa60ed68c2d3e7ec36fb193e2f3e984e8e4/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e4b00b084e325ae8d28ea86501d6c8cbb27ff2ffec60fa101c881d15ee9f02f3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d95c6648e5d07ba39cbf38e4723794ac6236f09001d8d368cadb76ef6a8e9086/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/f3438c49eade4e598f7b00c52b30ca344396401bf94da096d4535d49c49501ba/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1d37d4d8b7defc57a9237dc3aef70e62f5c4aeee5d94d7a9c7c887cde5cf9426/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/df17f4ad7faa0ac7625613f4ad5b8422a816a9d1e5efcd94f23802fc33fb6670/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1d4a2fa51b2917917dac2fa456d8009f390f074d35492ffa586df9f64058adbd/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/921323675d50550567123fdc2ccd5dbb22d318f6bdffdd04fbafe559a6c8b290/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/4657424315aed8e1592f230d4373b158dc515759be2f469665b6a93189f1fd3b/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/aaf549a22e12d10c422b9607fbbf945abf35fd194621cf50140b46d2c1bf178a/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/5d8de1bfac94b374f241792246ad881f40b3edaea8efca5804ce3376af0405b0/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/2ee359f972f95839346e9705d30cdfdd2e6e6aecec60c9ba4897926877a04a37/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/3e7584f0804a88c22add65a9e1099b62e8fee2b1fa5072e7001dc4d000d87819/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/5b9489d126690026ed964f797fa7a3f131d5934b0af314955460fa5cbb890912/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/ad3a1a8b182386a80b5c40d1e0165045ed8a8b6cee2422ddc8a97ed3d7daee0a/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/1363a0ecc630e0fcb3acf80a9342e6b7621c14205739fba01c9fe35b889cc87e/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/bcafaacf89271967284955cfb0f06defed1e4d5ad5838a026f1cd590d85e5c9e/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/39f8df1708c0b08647d42b2e96e87426150e328d899dad484025e31a64dd0441/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/c646b7d14608a6e39bdda233380031c8b8c0bdcd3d09412c3efb2268ac971588/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/35195c58e517987feb4e417a02bd804d75f47a6b86555d3c4dd91387a5f0cf92/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/89325daf39b3899d546a070fce5e5c2a1452354600b1f5479cd522a1219826cb/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/893c2a435ec1f80772983b31b8999ff3ae6353f43371e3dad5e3b9aeb8279bb2/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/3ee7297321783330707849a25bb166b6b40f6543100c717c7be5b48837876671/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/b9262a5913cd493a9dcb053373b6c63282018148b9402c46843f5ad508408627/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/c228d321385fdbbe5a70b28ada69310b4ed788bcab363697dafab257eaf95892/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/b3caf9ea38857219ac280cb43e741671569a65c792112739cffdc903a65b209b/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/0b94dbac760e4af2350b06b4c9e26a9c468c3213c3feae6025ccdc1b57a4cd3f/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/f936b0623ad44caa86e96778d7009fd8847f856677c9748720c9efe238573a2b/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/cfc7e9a46112b47596501cef6062185139cf0aab054062521499c0eee2123408/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/01e57df1fbd4ff47b61cc4f4fd4a2883033cc29e3ac8f01c78e5ffa577343cea/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/493e4e602fbd32981c94cf1b1423e6dbafde7f093758576ce56301281d9ce75c/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/4499176df579eb9b8ac318a1a8599836f843a071e0df3f89028cd9883ef84959/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/58a5525008ac5a578feae0961558ffebef96dd25e69d4c561a1beb111d91b77f/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/c4647c704bec748c3a5d6cb09250c797ab88207ff725a4efa4321a14651c2b4e/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/1419a8a452842abe94a8fada0b151c5e987c8c8db4e0f2edadc5d2412528a26c/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/e7029fb06ff093ba74c8b701fc33845327aa0711a4fd608754ed6a4f92b90004/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/e919b53eac9cdc68d4695d9417932a082158ff91acf3c465244b3ab23f7c0788/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/85448c08129c3bf341eb3a05ebc51896f56197a2fd6c1f75fda4cdf6538e9513/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/1d8862ba2c1efc4ac8e3a98acfde97cb016b43ee4a1cc0f6ee3a32f27f8ad6a8/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d315fc2e971d2ff0ab6fea0c8787043a06655b029c3273c6b09181ae110a01cd/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/922b867f27375a46ba459f608b730a3246903b62d361b8f4f70d3a0b881c7f34/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/881b25228efc1bbd9a7425c28d715b5c69ffc12683a445cedc370d0e2f987f99/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b15604b6bdf1200be487b84936d98af5a0853536807cc3abd0dc054a22f15d7f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/e2c39fd6e9a9cd1920aa2dcd71b5eae62056bb4ed346c88b0cb50d64cc33a296/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/3a7250e0feb75176f56d7d7d1ec4060bde74dbf74f78a916be131fd6c5929d6b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5d187eb2e8e378de05e875fbe5f4f8bed73063cafd21e09690c5d138d1586e05/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/6619534fdc42d8a4eae84f5c20eedb95e749f96ba1df6cfdb22bd6bcb14877b3/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.341428 5838 manager.go:217] Machine: {Timestamp:2026-10-30 16:20:38.34092206 +0000 UTC m=+0.089511437 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3118480 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d0d282908157e75aacc4e2f029971 SystemUUID:ec2d0d28-2908-157e-75aa-cc4e2f029971 BootID:59dcdc88-0b66-496a-b3ae-76cef16fb0fd Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddebdc11-43b3-4787-a3ff-bf95150f8d27/volumes/kubernetes.io~projected/kube-api-access-s6c2l DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/948f3d754808f828efa22bc27a3d7c31b701d85424db6ceb405de474567aaa84/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be9a1b81-45d0-46a0-a771-84d88046f28e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:234 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-151 DeviceMajor:0 DeviceMinor:151 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ca822f3-98ef-487f-a89d-1a1004beeda1/volumes/kubernetes.io~projected/kube-api-access-fdx66 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3d0efab9-17c2-4f08-9433-02670766f69c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62e319000ca88b93d246d429f68a234857e121cbee4eb5504346e3d0ba39c0b8/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13fe18553811e4ef9d4d2cd07996a2abb66728c7d29d2f1a67381e0009c5d1f6/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b75f55b1eabe52dfe9a1d14564c828142c3dae665cf2a30ff0180f333b3d3ab7/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e8b300bb7ba02af7d2ec639960543489b56a6fe8f316660cfca5d4edf70e072/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/be9a1b81-45d0-46a0-a771-84d88046f28e/volumes/kubernetes.io~projected/kube-api-access-6tn95 DeviceMajor:0 DeviceMinor:237 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e088f6ab1c408134be111533036761823177e97a6e9b9493acac296f80f53f2/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d47927b38cdc342416f26d788c9a3569434173c31a5a4746c2ee8b7bdd30fdd/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b/volumes/kubernetes.io~projected/kube-api-access-lwhlq DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dafd2b2cd7e4dfa578a58bf89762fedea6e1d7345757f3360da3e5bceb9e3f8/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4b13dfe3-9da5-4be8-9051-b1bbff1b083b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27fd84afe85bce8c2321e1b1b85b813f4c9ac089819b3c63c22e4e61bf8aea4a/userdata/shm DeviceMajor:0 DeviceMinor:257 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb2c72e6-372c-41c1-ba29-d47c0b521f34/volumes/kubernetes.io~projected/kube-api-access-99d2h DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5beb53b1-b44b-47eb-84d8-31660627d4a3/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/236ec382-dbd5-4974-9454-43b39e990c77/volumes/kubernetes.io~projected/kube-api-access-ln9pm DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5beb53b1-b44b-47eb-84d8-31660627d4a3/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:308 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37d8bfdd628d25d8061da4f3a87c1a8b41de644e58a636ec0551ec9cb59ac51f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6af2dcb8be6646a151b1dcbf23d69433191f530b283c91882258c5aa90bbd5b/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2196112-4441-49a3-b48f-1b80bf426e29/volumes/kubernetes.io~projected/kube-api-access-4qnf9 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e07b376-b6a8-4d05-b6f4-705695c0e92b/volumes/kubernetes.io~projected/kube-api-access-2jd74 DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22376cc752f611717ad034681e2aeb5629a1f1848734d9520754118aea8e962a/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3d0efab9-17c2-4f08-9433-02670766f69c/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2/volumes/kubernetes.io~projected/kube-api-access-qkz5f DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1796ff68883d583b3245e9a58383981660e62e063b39996985bf98ea72810169/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5aeb3fbd398c51d8dc64868da92878d26a596a7beb445896d953f1650f7cc0c1/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/67036dae-0e4a-473b-af97-e0ee20d824b8/volumes/kubernetes.io~projected/kube-api-access-57d5q DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81a6d8ed-431a-4b9b-bd73-d13b01208249/volumes/kubernetes.io~projected/kube-api-access-bk6g4 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be9a1b81-45d0-46a0-a771-84d88046f28e/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2228af531f82f28dc3beb9a5bb32696a044e1a6f85a29a9a873997054ee9940/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/236ec382-dbd5-4974-9454-43b39e990c77/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6b17f375c7f7270b0f267083307e091303bd28d80da37550d26258867925c02/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b36b8f12-981e-46a8-b422-632babc5d4a7/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:328 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db4b67644dc5923ad73e9ba0e1a4aa59668646dba0c302557567627c4838b297/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a2196112-4441-49a3-b48f-1b80bf426e29/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cd2b0c8e-8194-456d-8299-c43b1688404a/volumes/kubernetes.io~projected/kube-api-access-cblkc DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/791cb17f-1c21-414c-9f90-196b26aa3570/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/791cb17f-1c21-414c-9f90-196b26aa3570/volumes/kubernetes.io~projected/kube-api-access-p8zmd DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a2196112-4441-49a3-b48f-1b80bf426e29/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5beb53b1-b44b-47eb-84d8-31660627d4a3/volumes/kubernetes.io~projected/kube-api-access-xpkkc DeviceMajor:0 DeviceMinor:316 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b13dfe3-9da5-4be8-9051-b1bbff1b083b/volumes/kubernetes.io~projected/kube-api-access-2rbfc DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d0efab9-17c2-4f08-9433-02670766f69c/volumes/kubernetes.io~projected/kube-api-access-4n7bv DeviceMajor:0 DeviceMinor:231 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddebdc11-43b3-4787-a3ff-bf95150f8d27/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:323 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/066e878b8059c156ab32b4d15b8589b9e29037a987260ef92e01207dd3249417/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5beb53b1-b44b-47eb-84d8-31660627d4a3/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:300 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a65ebc67454e0ccf639133a0a0dcb08ca5b36c47b5434e35bbae75bed1115da/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30634e3d-5f01-4b9b-88ad-03893a2b444e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fcc8b41f6f78812d8fd579ca43e70a488b6f9659ef2efa107fdef1be43b3c4d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ca822f3-98ef-487f-a89d-1a1004beeda1/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4b13dfe3-9da5-4be8-9051-b1bbff1b083b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d90a91c8-f7a6-4526-a13c-ce484713b6b5/volumes/kubernetes.io~projected/kube-api-access-8skbh DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:066e878b8059c15 MacAddress:12:10:ab:9d:90:4e Speed:10000 Mtu:8901} {Name:1796ff68883d583 MacAddress:ea:bf:08:e1:9b:2b Speed:10000 Mtu:8901} {Name:2e8b300bb7ba02a MacAddress:22:0a:d6:e8:25:1f Speed:10000 Mtu:8901} {Name:3d47927b38cdc34 MacAddress:b6:43:1a:0a:10:a1 Speed:10000 Mtu:8901} {Name:5e088f6ab1c4081 MacAddress:9a:d4:bf:68:a9:b9 Speed:10000 Mtu:8901} {Name:8dafd2b2cd7e4df MacAddress:f6:ca:9f:f1:ca:33 Speed:10000 Mtu:8901} {Name:9a65ebc67454e0c MacAddress:56:74:14:15:8c:fa Speed:10000 Mtu:8901} {Name:a6af2dcb8be6646 MacAddress:ae:78:26:a0:88:33 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:de:56:ba:8b Speed:0 Mtu:9001} {Name:br-int MacAddress:e2:18:df:6a:00:fc Speed:0 Mtu:8901} {Name:c2228af531f82f2 MacAddress:12:5c:1d:45:39:de Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:de:56:ba:8b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:96:f0:ca:2e:c3:fd Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:47:9d:fc:9c:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.341732 5838 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.341775 5838 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342076 5838 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342102 5838 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-219.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342230 5838 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342238 5838 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342248 5838 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342268 5838 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342361 5838 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342512 5838 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342630 5838 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342664 5838 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342673 5838 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342684 5838 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342692 5838 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.342700 5838 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.343397 5838 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.343407 5838 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.343870 5838 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346109 5838 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346119 5838 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346212 5838 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346399 5838 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346426 5838 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346513 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346526 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346531 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346536 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346542 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346548 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346554 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346559 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346567 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346573 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346581 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346591 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346624 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.346630 5838 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.349919 5838 server.go:1263] "Started kubelet" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.349990 5838 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.350031 5838 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.350110 5838 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.350438 5838 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:20:38.354724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.351041 5838 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:20:38.359142 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.358423 5838 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:20:38.359142 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.358452 5838 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:20:38.359142 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.358508 5838 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:20:38.359142 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.358508 5838 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:14:28 +0000 UTC" deadline="2028-04-27 09:48:40.457566402 +0000 UTC" Oct 30 16:20:38.359142 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.358527 5838 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13073h28m2.099042715s" Oct 30 16:20:38.359142 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:20:38.358540 5838 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-219.ec2.internal\" not found" Oct 30 16:20:38.359142 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.359071 5838 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:20:38.360809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.358520 5838 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:20:38.360809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.360524 5838 factory.go:153] Registering CRI-O factory Oct 30 16:20:38.360809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.360544 5838 factory.go:223] Registration of the crio container factory successfully Oct 30 16:20:38.360809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.360599 5838 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:20:38.360809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.360609 5838 factory.go:55] Registering systemd factory Oct 30 16:20:38.360809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.360617 5838 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:20:38.360809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.360637 5838 factory.go:103] Registering Raw factory Oct 30 16:20:38.360809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.360652 5838 manager.go:1199] Started watching for new ooms in manager Oct 30 16:20:38.361324 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.361055 5838 manager.go:319] Starting recovery of all containers Oct 30 16:20:38.362510 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362456 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae1acb11-b25d-43d5-b027-5edcf8e5fd1b" volumeName="kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-tls" seLinuxMountContext="" Oct 30 16:20:38.362600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362512 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5beb53b1-b44b-47eb-84d8-31660627d4a3" volumeName="kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-client-ca-bundle" seLinuxMountContext="" Oct 30 16:20:38.362600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362528 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236ec382-dbd5-4974-9454-43b39e990c77" volumeName="kubernetes.io/secret/236ec382-dbd5-4974-9454-43b39e990c77-cert" seLinuxMountContext="" Oct 30 16:20:38.362600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362542 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae1acb11-b25d-43d5-b027-5edcf8e5fd1b" volumeName="kubernetes.io/configmap/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 16:20:38.362600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362554 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d0efab9-17c2-4f08-9433-02670766f69c" volumeName="kubernetes.io/secret/3d0efab9-17c2-4f08-9433-02670766f69c-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 16:20:38.362600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362567 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e07b376-b6a8-4d05-b6f4-705695c0e92b" volumeName="kubernetes.io/projected/3e07b376-b6a8-4d05-b6f4-705695c0e92b-kube-api-access-2jd74" seLinuxMountContext="" Oct 30 16:20:38.362600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362578 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2" volumeName="kubernetes.io/projected/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-kube-api-access-qkz5f" seLinuxMountContext="" Oct 30 16:20:38.362600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362591 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5beb53b1-b44b-47eb-84d8-31660627d4a3" volumeName="kubernetes.io/projected/5beb53b1-b44b-47eb-84d8-31660627d4a3-kube-api-access-xpkkc" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362607 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be9a1b81-45d0-46a0-a771-84d88046f28e" volumeName="kubernetes.io/configmap/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362620 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddebdc11-43b3-4787-a3ff-bf95150f8d27" volumeName="kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362633 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67036dae-0e4a-473b-af97-e0ee20d824b8" volumeName="kubernetes.io/configmap/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-daemon-config" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362657 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67036dae-0e4a-473b-af97-e0ee20d824b8" volumeName="kubernetes.io/projected/67036dae-0e4a-473b-af97-e0ee20d824b8-kube-api-access-57d5q" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362669 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be9a1b81-45d0-46a0-a771-84d88046f28e" volumeName="kubernetes.io/projected/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-api-access-6tn95" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362682 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ca822f3-98ef-487f-a89d-1a1004beeda1" volumeName="kubernetes.io/secret/9ca822f3-98ef-487f-a89d-1a1004beeda1-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362702 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb2c72e6-372c-41c1-ba29-d47c0b521f34" volumeName="kubernetes.io/projected/fb2c72e6-372c-41c1-ba29-d47c0b521f34-kube-api-access-99d2h" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362717 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30634e3d-5f01-4b9b-88ad-03893a2b444e" volumeName="kubernetes.io/secret/30634e3d-5f01-4b9b-88ad-03893a2b444e-agent-certs" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362728 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d0efab9-17c2-4f08-9433-02670766f69c" volumeName="kubernetes.io/secret/3d0efab9-17c2-4f08-9433-02670766f69c-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362741 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5beb53b1-b44b-47eb-84d8-31660627d4a3" volumeName="kubernetes.io/configmap/5beb53b1-b44b-47eb-84d8-31660627d4a3-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362754 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5beb53b1-b44b-47eb-84d8-31660627d4a3" volumeName="kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362767 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81a6d8ed-431a-4b9b-bd73-d13b01208249" volumeName="kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362779 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81a6d8ed-431a-4b9b-bd73-d13b01208249" volumeName="kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-cni-binary-copy" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362797 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae1acb11-b25d-43d5-b027-5edcf8e5fd1b" volumeName="kubernetes.io/configmap/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-metrics-client-ca" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362809 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be9a1b81-45d0-46a0-a771-84d88046f28e" volumeName="kubernetes.io/secret/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362822 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b13dfe3-9da5-4be8-9051-b1bbff1b083b" volumeName="kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-env-overrides" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362834 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67036dae-0e4a-473b-af97-e0ee20d824b8" volumeName="kubernetes.io/configmap/67036dae-0e4a-473b-af97-e0ee20d824b8-cni-binary-copy" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362849 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81a6d8ed-431a-4b9b-bd73-d13b01208249" volumeName="kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362860 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81a6d8ed-431a-4b9b-bd73-d13b01208249" volumeName="kubernetes.io/projected/81a6d8ed-431a-4b9b-bd73-d13b01208249-kube-api-access-bk6g4" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362872 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be9a1b81-45d0-46a0-a771-84d88046f28e" volumeName="kubernetes.io/secret/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362890 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd2b0c8e-8194-456d-8299-c43b1688404a" volumeName="kubernetes.io/projected/cd2b0c8e-8194-456d-8299-c43b1688404a-kube-api-access-cblkc" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362902 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236ec382-dbd5-4974-9454-43b39e990c77" volumeName="kubernetes.io/projected/236ec382-dbd5-4974-9454-43b39e990c77-kube-api-access-ln9pm" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362932 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="791cb17f-1c21-414c-9f90-196b26aa3570" volumeName="kubernetes.io/empty-dir/791cb17f-1c21-414c-9f90-196b26aa3570-tmp-dir" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362947 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ca822f3-98ef-487f-a89d-1a1004beeda1" volumeName="kubernetes.io/projected/9ca822f3-98ef-487f-a89d-1a1004beeda1-kube-api-access-fdx66" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362959 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae1acb11-b25d-43d5-b027-5edcf8e5fd1b" volumeName="kubernetes.io/empty-dir/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-textfile" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362985 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b36b8f12-981e-46a8-b422-632babc5d4a7" volumeName="kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.362998 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d0efab9-17c2-4f08-9433-02670766f69c" volumeName="kubernetes.io/projected/3d0efab9-17c2-4f08-9433-02670766f69c-kube-api-access-4n7bv" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363010 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5beb53b1-b44b-47eb-84d8-31660627d4a3" volumeName="kubernetes.io/empty-dir/5beb53b1-b44b-47eb-84d8-31660627d4a3-audit-log" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363034 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="791cb17f-1c21-414c-9f90-196b26aa3570" volumeName="kubernetes.io/projected/791cb17f-1c21-414c-9f90-196b26aa3570-kube-api-access-p8zmd" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363048 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae1acb11-b25d-43d5-b027-5edcf8e5fd1b" volumeName="kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363060 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be9a1b81-45d0-46a0-a771-84d88046f28e" volumeName="kubernetes.io/empty-dir/be9a1b81-45d0-46a0-a771-84d88046f28e-volume-directive-shadow" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363072 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30634e3d-5f01-4b9b-88ad-03893a2b444e" volumeName="kubernetes.io/configmap/30634e3d-5f01-4b9b-88ad-03893a2b444e-konnectivity-ca" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363084 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d0efab9-17c2-4f08-9433-02670766f69c" volumeName="kubernetes.io/configmap/3d0efab9-17c2-4f08-9433-02670766f69c-metrics-client-ca" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363096 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e07b376-b6a8-4d05-b6f4-705695c0e92b" volumeName="kubernetes.io/configmap/3e07b376-b6a8-4d05-b6f4-705695c0e92b-iptables-alerter-script" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363109 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5beb53b1-b44b-47eb-84d8-31660627d4a3" volumeName="kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363123 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="791cb17f-1c21-414c-9f90-196b26aa3570" volumeName="kubernetes.io/secret/791cb17f-1c21-414c-9f90-196b26aa3570-metrics-tls" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363135 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b13dfe3-9da5-4be8-9051-b1bbff1b083b" volumeName="kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovnkube-script-lib" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363148 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b13dfe3-9da5-4be8-9051-b1bbff1b083b" volumeName="kubernetes.io/projected/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-kube-api-access-2rbfc" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363161 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="791cb17f-1c21-414c-9f90-196b26aa3570" volumeName="kubernetes.io/configmap/791cb17f-1c21-414c-9f90-196b26aa3570-config-volume" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363173 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ca822f3-98ef-487f-a89d-1a1004beeda1" volumeName="kubernetes.io/empty-dir/9ca822f3-98ef-487f-a89d-1a1004beeda1-data-volume" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363185 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2196112-4441-49a3-b48f-1b80bf426e29" volumeName="kubernetes.io/empty-dir/a2196112-4441-49a3-b48f-1b80bf426e29-etc-tuned" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363198 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d90a91c8-f7a6-4526-a13c-ce484713b6b5" volumeName="kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363210 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b13dfe3-9da5-4be8-9051-b1bbff1b083b" volumeName="kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovnkube-config" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363224 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b13dfe3-9da5-4be8-9051-b1bbff1b083b" volumeName="kubernetes.io/secret/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363237 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2196112-4441-49a3-b48f-1b80bf426e29" volumeName="kubernetes.io/projected/a2196112-4441-49a3-b48f-1b80bf426e29-kube-api-access-4qnf9" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363249 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae1acb11-b25d-43d5-b027-5edcf8e5fd1b" volumeName="kubernetes.io/projected/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-kube-api-access-lwhlq" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363263 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb2c72e6-372c-41c1-ba29-d47c0b521f34" volumeName="kubernetes.io/configmap/fb2c72e6-372c-41c1-ba29-d47c0b521f34-serviceca" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363277 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be9a1b81-45d0-46a0-a771-84d88046f28e" volumeName="kubernetes.io/configmap/be9a1b81-45d0-46a0-a771-84d88046f28e-metrics-client-ca" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363300 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddebdc11-43b3-4787-a3ff-bf95150f8d27" volumeName="kubernetes.io/projected/ddebdc11-43b3-4787-a3ff-bf95150f8d27-kube-api-access-s6c2l" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363313 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd2b0c8e-8194-456d-8299-c43b1688404a" volumeName="kubernetes.io/empty-dir/cd2b0c8e-8194-456d-8299-c43b1688404a-tmp-dir" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363327 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ca822f3-98ef-487f-a89d-1a1004beeda1" volumeName="kubernetes.io/configmap/9ca822f3-98ef-487f-a89d-1a1004beeda1-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363338 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5beb53b1-b44b-47eb-84d8-31660627d4a3" volumeName="kubernetes.io/configmap/5beb53b1-b44b-47eb-84d8-31660627d4a3-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363350 5838 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2196112-4441-49a3-b48f-1b80bf426e29" volumeName="kubernetes.io/empty-dir/a2196112-4441-49a3-b48f-1b80bf426e29-tmp" seLinuxMountContext="" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363362 5838 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:20:38.364319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.363370 5838 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:20:38.377317 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.377283 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:20:38.382645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.382618 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:20:38.387190 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.386153 5838 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:20:38.387571 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.387550 5838 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:20:38.387685 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.387575 5838 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:20:38.387685 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.387595 5838 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:20:38.387685 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:20:38.387635 5838 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 16:20:38.404413 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.404381 5838 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:38.404554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.404466 5838 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 16:20:38.405915 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.405897 5838 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:38.407417 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.407398 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:20:38.407749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.407720 5838 generic.go:356] "Generic (PLEG): container finished" podID="4b13dfe3-9da5-4be8-9051-b1bbff1b083b" containerID="01a36d92f8a460b4854691eaefe1f298a98470c76e2bdce84461014e50429425" exitCode=1 Oct 30 16:20:38.409623 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.409594 5838 generic.go:356] "Generic (PLEG): container finished" podID="823ffa96cf0d2439ccc4c6edac7c16b5" containerID="af1e1b931cbd4a97cce4f14bd0bfb9246e1a7f2e5a66135c1b6dba05de613bf6" exitCode=0 Oct 30 16:20:38.418203 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.418179 5838 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:38.423886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.423854 5838 generic.go:356] "Generic (PLEG): container finished" podID="81a6d8ed-431a-4b9b-bd73-d13b01208249" containerID="23e798cd3d296ca0cf00b0e8537d106877e72078d221be67ca8a78412b30e120" exitCode=0 Oct 30 16:20:38.423886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.423883 5838 generic.go:356] "Generic (PLEG): container finished" podID="81a6d8ed-431a-4b9b-bd73-d13b01208249" containerID="023946bfd7db7d3cfbd325d974b363ade09bfe7d953089c4e817e48ec9b4f412" exitCode=0 Oct 30 16:20:38.423886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.423891 5838 generic.go:356] "Generic (PLEG): container finished" podID="81a6d8ed-431a-4b9b-bd73-d13b01208249" containerID="8a6b138d39000d13a0ef795b896f63fe0d41b75419b9d89d4ab60f8bcbbcc168" exitCode=0 Oct 30 16:20:38.424180 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.423899 5838 generic.go:356] "Generic (PLEG): container finished" podID="81a6d8ed-431a-4b9b-bd73-d13b01208249" containerID="5c13a014aed4350aa3eecd49f4ad82f35b7eb4bae3ae8dd75fb42ae63e0662db" exitCode=0 Oct 30 16:20:38.424180 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.423906 5838 generic.go:356] "Generic (PLEG): container finished" podID="81a6d8ed-431a-4b9b-bd73-d13b01208249" containerID="927ba85ddd325374760076133f8f1fb6a3f8e49cc13a875710cf66659d9ccb60" exitCode=0 Oct 30 16:20:38.424180 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.423913 5838 generic.go:356] "Generic (PLEG): container finished" podID="81a6d8ed-431a-4b9b-bd73-d13b01208249" containerID="aee26331736e802e4da5198b40f5a7a5a961a85750f6017a2724089e33641e12" exitCode=0 Oct 30 16:20:38.427425 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.427404 5838 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:38.430384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.430358 5838 generic.go:356] "Generic (PLEG): container finished" podID="ae1acb11-b25d-43d5-b027-5edcf8e5fd1b" containerID="e4fa9e4166c47be78cfa740c9def8c166c5ac41127e2c0b7a95e90221c746324" exitCode=0 Oct 30 16:20:38.453493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.453432 5838 manager.go:324] Recovery completed Oct 30 16:20:38.476158 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476131 5838 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:20:38.476158 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476150 5838 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:20:38.476341 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476172 5838 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:20:38.476341 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476311 5838 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 16:20:38.476341 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476321 5838 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 16:20:38.476341 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476338 5838 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 16:20:38.476341 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476343 5838 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 16:20:38.476476 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476349 5838 policy_none.go:50] "Start" Oct 30 16:20:38.476476 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476357 5838 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:20:38.476476 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476365 5838 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:20:38.476558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476503 5838 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 16:20:38.476558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476512 5838 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 16:20:38.476558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.476518 5838 policy_none.go:44] "Start" Oct 30 16:20:38.480880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.480849 5838 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:20:38.484398 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.484384 5838 manager.go:341] "Starting Device Plugin manager" Oct 30 16:20:38.485105 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:20:38.484412 5838 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:20:38.485145 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.485111 5838 server.go:87] "Starting device plugin registration server" Oct 30 16:20:38.485459 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.485446 5838 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:20:38.485493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.485461 5838 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:20:38.485555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.485541 5838 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:20:38.485661 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.485628 5838 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:20:38.485736 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.485668 5838 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:20:38.485736 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.485672 5838 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:20:38.485809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.485716 5838 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:20:38.48565888Z","UUID":"e6f0a64e-2018-4720-8a17-8143cb357e22","Handler":null,"Name":"","Endpoint":""} Oct 30 16:20:38.487126 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.487106 5838 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:20:38.487224 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.487131 5838 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:20:38.487333 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:20:38.487318 5838 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:20:38.487742 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.487705 5838 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal"] Oct 30 16:20:38.488514 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.488476 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" event={"ID":"823ffa96cf0d2439ccc4c6edac7c16b5","Type":"ContainerStarted","Data":"03dd121c72aab9cd33a9ad8730fe5e799e78a5e5640471b84eeb5c0a1105871d"} Oct 30 16:20:38.488633 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.488526 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" event={"ID":"823ffa96cf0d2439ccc4c6edac7c16b5","Type":"ContainerDied","Data":"af1e1b931cbd4a97cce4f14bd0bfb9246e1a7f2e5a66135c1b6dba05de613bf6"} Oct 30 16:20:38.488633 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.488548 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" event={"ID":"823ffa96cf0d2439ccc4c6edac7c16b5","Type":"ContainerStarted","Data":"62e319000ca88b93d246d429f68a234857e121cbee4eb5504346e3d0ba39c0b8"} Oct 30 16:20:38.488633 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.488591 5838 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.488633 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.488618 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" event={"ID":"a8e5ad7f7f3345c1093a31c3b5fc9d86","Type":"ContainerStarted","Data":"1e52594d2108310a8320eb6a6b30fcebba9d8dbc3fdb4cd6bed225dee14d3ec5"} Oct 30 16:20:38.488822 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.488640 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" event={"ID":"a8e5ad7f7f3345c1093a31c3b5fc9d86","Type":"ContainerStarted","Data":"5aeb3fbd398c51d8dc64868da92878d26a596a7beb445896d953f1650f7cc0c1"} Oct 30 16:20:38.488822 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.488642 5838 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.510460 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.510435 5838 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:38.510593 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:20:38.510485 5838 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-219.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.511129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.511113 5838 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:38.511215 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:20:38.511158 5838 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.585769 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.585735 5838 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:38.587727 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.587704 5838 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:38.587828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.587742 5838 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:38.587828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.587753 5838 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:38.587828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.587789 5838 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.605468 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.605440 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8e5ad7f7f3345c1093a31c3b5fc9d86-config\") pod \"kube-apiserver-proxy-ip-10-0-137-219.ec2.internal\" (UID: \"a8e5ad7f7f3345c1093a31c3b5fc9d86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.605575 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.605472 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/823ffa96cf0d2439ccc4c6edac7c16b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"823ffa96cf0d2439ccc4c6edac7c16b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.605575 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.605494 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/823ffa96cf0d2439ccc4c6edac7c16b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"823ffa96cf0d2439ccc4c6edac7c16b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.621457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.621432 5838 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.621564 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.621498 5838 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.705924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.705842 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/823ffa96cf0d2439ccc4c6edac7c16b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"823ffa96cf0d2439ccc4c6edac7c16b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.705924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.705876 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/823ffa96cf0d2439ccc4c6edac7c16b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"823ffa96cf0d2439ccc4c6edac7c16b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.705924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.705897 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8e5ad7f7f3345c1093a31c3b5fc9d86-config\") pod \"kube-apiserver-proxy-ip-10-0-137-219.ec2.internal\" (UID: \"a8e5ad7f7f3345c1093a31c3b5fc9d86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.706131 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.705944 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8e5ad7f7f3345c1093a31c3b5fc9d86-config\") pod \"kube-apiserver-proxy-ip-10-0-137-219.ec2.internal\" (UID: \"a8e5ad7f7f3345c1093a31c3b5fc9d86\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.706131 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.705957 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/823ffa96cf0d2439ccc4c6edac7c16b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"823ffa96cf0d2439ccc4c6edac7c16b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:20:38.706131 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:38.705957 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/823ffa96cf0d2439ccc4c6edac7c16b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"823ffa96cf0d2439ccc4c6edac7c16b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:20:39.343416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.343372 5838 apiserver.go:52] "Watching apiserver" Oct 30 16:20:39.381921 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.381895 5838 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:20:39.382432 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.382412 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-4856h","kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal","openshift-monitoring/metrics-server-5d79986586-bccmm","openshift-image-registry/node-ca-46t9k","openshift-ingress-canary/ingress-canary-csxp6","openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5","openshift-multus/multus-qdzgt","openshift-network-operator/iptables-alerter-c2tkd","openshift-dns/dns-default-vp4m7","openshift-network-diagnostics/network-check-target-mv4lc","openshift-insights/insights-runtime-extractor-n8h27","openshift-monitoring/node-exporter-b9fbq","kube-system/konnectivity-agent-h8rzc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv","openshift-multus/multus-additional-cni-plugins-vcfvx","openshift-ovn-kubernetes/ovnkube-node-v4lzf","openshift-cluster-node-tuning-operator/tuned-4g45x","openshift-dns/node-resolver-7rnq2","kube-system/global-pull-secret-syncer-lt2pz","openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47"] Oct 30 16:20:39.397720 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.397671 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:20:39.398212 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.398178 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:20:39.398343 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.398195 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:20:39.398416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.398395 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:20:39.398587 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.398568 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:20:39.398800 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.398746 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:20:39.398939 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.398917 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:20:39.399026 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.398782 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:20:39.399402 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.399382 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:20:39.399569 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.399547 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8fzhw\"" Oct 30 16:20:39.399837 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.399806 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dlk52\"" Oct 30 16:20:39.399933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.399918 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bhx7q\"" Oct 30 16:20:39.407407 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.407386 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:20:39.407581 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.407483 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:20:39.407581 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.407554 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:20:39.407581 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.407577 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j7zxr\"" Oct 30 16:20:39.408039 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408012 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:20:39.408112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408098 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-t25rz\"" Oct 30 16:20:39.408223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408207 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-snnbg\"" Oct 30 16:20:39.408286 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408268 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:20:39.408336 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408297 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 16:20:39.408416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408397 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:20:39.408466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408427 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:39.408466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408440 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:20:39.408540 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408463 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:20:39.408589 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408572 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2smlp\"" Oct 30 16:20:39.408640 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408603 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v666l\"" Oct 30 16:20:39.408640 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408606 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 16:20:39.408640 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408604 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 16:20:39.408640 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408637 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:20:39.408768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408716 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:20:39.408768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408721 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:20:39.408924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408772 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-svxjg\"" Oct 30 16:20:39.408924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408793 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8qclp\"" Oct 30 16:20:39.408924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408804 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:20:39.408924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408817 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:20:39.408924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408821 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mzplq\"" Oct 30 16:20:39.408924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408838 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:20:39.408924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408857 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zr2xc\"" Oct 30 16:20:39.408924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408890 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:20:39.408924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408905 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 16:20:39.408924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408906 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 16:20:39.409283 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408937 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:20:39.409283 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.408966 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:20:39.409283 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409043 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cpfx5\"" Oct 30 16:20:39.409283 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409070 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 16:20:39.409283 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409248 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:20:39.409283 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409263 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:20:39.409479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409302 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:20:39.409479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409311 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-g5m2c\"" Oct 30 16:20:39.409479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409334 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fjt7p\"" Oct 30 16:20:39.409479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409426 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 16:20:39.409479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409466 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:39.409645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409529 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-85pv7n2is4r7c\"" Oct 30 16:20:39.409645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409613 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:20:39.409734 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409687 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:20:39.409787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409756 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:20:39.410000 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.409985 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2ph4\"" Oct 30 16:20:39.410033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.410016 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:39.410114 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.410100 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:39.410165 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.410120 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:20:39.410165 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.410124 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:20:39.410303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.410291 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 16:20:39.410662 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.410644 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:20:39.410730 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.410687 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 16:20:39.410781 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.410739 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 16:20:39.410837 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.410789 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lk6j9\"" Oct 30 16:20:39.410837 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.410801 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 16:20:39.410837 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.410811 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:20:39.411016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.411001 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ltrwz\"" Oct 30 16:20:39.411079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.411004 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:20:39.411079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.411010 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 16:20:39.411079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.411028 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6x9tv\"" Oct 30 16:20:39.411079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.411053 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:20:39.411195 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.411114 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 16:20:39.440836 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.440806 5838 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:20:39.440986 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.440874 5838 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:20:39.460462 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.460439 5838 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:20:39.463097 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.463075 5838 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:39.463186 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:20:39.463120 5838 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-219.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Oct 30 16:20:39.463768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.463753 5838 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:39.463821 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:20:39.463785 5838 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Oct 30 16:20:39.509132 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509100 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysconfig\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.509274 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509144 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5beb53b1-b44b-47eb-84d8-31660627d4a3-metrics-server-audit-profiles\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.509274 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509173 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:39.509274 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509232 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:39.509435 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509380 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-systemd\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.509435 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509422 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tn95\" (UniqueName: \"kubernetes.io/projected/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-api-access-6tn95\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:39.509535 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509456 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-metrics-client-ca\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.509535 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509486 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:39.509634 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509580 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5beb53b1-b44b-47eb-84d8-31660627d4a3-metrics-server-audit-profiles\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.509634 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509591 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2196112-4441-49a3-b48f-1b80bf426e29-tmp\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.509724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509643 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rbfc\" (UniqueName: \"kubernetes.io/projected/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-kube-api-access-2rbfc\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.509724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509657 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2196112-4441-49a3-b48f-1b80bf426e29-tmp\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.509724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509684 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.509724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509702 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-metrics-client-ca\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.509724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509711 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jd74\" (UniqueName: \"kubernetes.io/projected/3e07b376-b6a8-4d05-b6f4-705695c0e92b-kube-api-access-2jd74\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:20:39.509949 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509752 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-env-overrides\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.509949 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509780 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:20:39.509949 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509819 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-client-ca-bundle\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.509949 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509854 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-system-cni-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.509949 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509886 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57d5q\" (UniqueName: \"kubernetes.io/projected/67036dae-0e4a-473b-af97-e0ee20d824b8-kube-api-access-57d5q\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.509949 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509911 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6c2l\" (UniqueName: \"kubernetes.io/projected/ddebdc11-43b3-4787-a3ff-bf95150f8d27-kube-api-access-s6c2l\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:20:39.509949 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509938 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ca822f3-98ef-487f-a89d-1a1004beeda1-crio-socket\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:39.510221 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.509958 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-env-overrides\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.510221 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510080 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-client-ca-bundle\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.510221 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510085 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:39.510379 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510359 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-hostroot\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.510412 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510394 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-multus-certs\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.510441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510413 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdx66\" (UniqueName: \"kubernetes.io/projected/9ca822f3-98ef-487f-a89d-1a1004beeda1-kube-api-access-fdx66\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:39.510441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510432 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-tls\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.510502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510464 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddebdc11-43b3-4787-a3ff-bf95150f8d27-metrics-certs\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:20:39.510502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510473 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-cni-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.510571 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510511 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-daemon-config\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.510571 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510540 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb2c72e6-372c-41c1-ba29-d47c0b521f34-host\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:20:39.510571 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510565 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99d2h\" (UniqueName: \"kubernetes.io/projected/fb2c72e6-372c-41c1-ba29-d47c0b521f34-kube-api-access-99d2h\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:20:39.510705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510590 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a2196112-4441-49a3-b48f-1b80bf426e29-etc-tuned\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.510705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510630 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-socket-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.510705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510663 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-tls\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.510705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510676 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a2196112-4441-49a3-b48f-1b80bf426e29-etc-tuned\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.510705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510665 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwhlq\" (UniqueName: \"kubernetes.io/projected/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-kube-api-access-lwhlq\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.510896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510724 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/791cb17f-1c21-414c-9f90-196b26aa3570-config-volume\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:39.510896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510753 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/791cb17f-1c21-414c-9f90-196b26aa3570-metrics-tls\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:39.510896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510727 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-daemon-config\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.510896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510779 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkz5f\" (UniqueName: \"kubernetes.io/projected/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-kube-api-access-qkz5f\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.510896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510807 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca822f3-98ef-487f-a89d-1a1004beeda1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:39.510896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510838 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-etc-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.510896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510862 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791cb17f-1c21-414c-9f90-196b26aa3570-tmp-dir\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:39.511183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510918 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/791cb17f-1c21-414c-9f90-196b26aa3570-config-volume\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:39.511183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510961 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791cb17f-1c21-414c-9f90-196b26aa3570-tmp-dir\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:39.511183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.510998 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/791cb17f-1c21-414c-9f90-196b26aa3570-metrics-tls\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:39.511183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511014 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-secret-metrics-server-tls\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.511183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511048 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d0efab9-17c2-4f08-9433-02670766f69c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:39.511183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511058 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9ca822f3-98ef-487f-a89d-1a1004beeda1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:39.511183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511083 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d0efab9-17c2-4f08-9433-02670766f69c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:39.511183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511111 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-modprobe-d\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.511183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511133 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-slash\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.511183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511175 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511186 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-secret-metrics-server-tls\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511204 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0efab9-17c2-4f08-9433-02670766f69c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511252 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d0efab9-17c2-4f08-9433-02670766f69c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511290 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b36b8f12-981e-46a8-b422-632babc5d4a7-dbus\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511307 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d0efab9-17c2-4f08-9433-02670766f69c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511315 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-kubelet\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511345 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-cnibin\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511383 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovn-node-metrics-cert\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511387 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d0efab9-17c2-4f08-9433-02670766f69c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511452 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-kubelet\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511489 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-etc-kubernetes\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511515 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-node-log\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.511596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511572 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-sys-fs\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511594 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovn-node-metrics-cert\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511625 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511664 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-conf-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511690 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b36b8f12-981e-46a8-b422-632babc5d4a7-kubelet-config\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511720 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd2b0c8e-8194-456d-8299-c43b1688404a-hosts-file\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511752 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7bv\" (UniqueName: \"kubernetes.io/projected/3d0efab9-17c2-4f08-9433-02670766f69c-kube-api-access-4n7bv\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511768 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511799 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysctl-conf\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511870 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpkkc\" (UniqueName: \"kubernetes.io/projected/5beb53b1-b44b-47eb-84d8-31660627d4a3-kube-api-access-xpkkc\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511942 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.511952 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/be9a1b81-45d0-46a0-a771-84d88046f28e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512094 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512119 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/be9a1b81-45d0-46a0-a771-84d88046f28e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:39.512637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512153 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.512637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512244 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.512637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512284 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln9pm\" (UniqueName: \"kubernetes.io/projected/236ec382-dbd5-4974-9454-43b39e990c77-kube-api-access-ln9pm\") pod \"ingress-canary-csxp6\" (UID: \"236ec382-dbd5-4974-9454-43b39e990c77\") " pod="openshift-ingress-canary/ingress-canary-csxp6" Oct 30 16:20:39.512637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512307 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-cni-binary-copy\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.512637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512313 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-root\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.512637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512343 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67036dae-0e4a-473b-af97-e0ee20d824b8-cni-binary-copy\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.512637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512365 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-run\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.512637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512409 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qnf9\" (UniqueName: \"kubernetes.io/projected/a2196112-4441-49a3-b48f-1b80bf426e29-kube-api-access-4qnf9\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.512637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512437 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-device-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.512637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512537 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-cni-netd\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.512637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512547 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67036dae-0e4a-473b-af97-e0ee20d824b8-cni-binary-copy\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.512637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512574 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk6g4\" (UniqueName: \"kubernetes.io/projected/81a6d8ed-431a-4b9b-bd73-d13b01208249-kube-api-access-bk6g4\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512650 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e07b376-b6a8-4d05-b6f4-705695c0e92b-iptables-alerter-script\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512682 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-wtmp\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512703 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-sys\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512743 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-var-lib-kubelet\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512769 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-var-lib-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512810 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-cni-bin\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512849 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5beb53b1-b44b-47eb-84d8-31660627d4a3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512872 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e07b376-b6a8-4d05-b6f4-705695c0e92b-iptables-alerter-script\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512903 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skbh\" (UniqueName: \"kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh\") pod \"network-check-target-mv4lc\" (UID: \"d90a91c8-f7a6-4526-a13c-ce484713b6b5\") " pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512939 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-netns\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.512992 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30634e3d-5f01-4b9b-88ad-03893a2b444e-agent-certs\") pod \"konnectivity-agent-h8rzc\" (UID: \"30634e3d-5f01-4b9b-88ad-03893a2b444e\") " pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513019 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-lib-modules\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513020 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5beb53b1-b44b-47eb-84d8-31660627d4a3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513042 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e07b376-b6a8-4d05-b6f4-705695c0e92b-host-slash\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513079 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-sys\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513131 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-k8s-cni-cncf-io\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.513143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513154 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-cni-bin\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513199 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysctl-d\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513231 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovnkube-config\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513246 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-kubernetes\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513255 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30634e3d-5f01-4b9b-88ad-03893a2b444e-agent-certs\") pod \"konnectivity-agent-h8rzc\" (UID: \"30634e3d-5f01-4b9b-88ad-03893a2b444e\") " pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513293 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-run-netns\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513374 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-cnibin\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513402 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovnkube-config\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513429 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca822f3-98ef-487f-a89d-1a1004beeda1-data-volume\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513469 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513500 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30634e3d-5f01-4b9b-88ad-03893a2b444e-konnectivity-ca\") pod \"konnectivity-agent-h8rzc\" (UID: \"30634e3d-5f01-4b9b-88ad-03893a2b444e\") " pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513519 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-systemd\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513570 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5beb53b1-b44b-47eb-84d8-31660627d4a3-audit-log\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513599 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513620 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca822f3-98ef-487f-a89d-1a1004beeda1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513620 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ca822f3-98ef-487f-a89d-1a1004beeda1-data-volume\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513639 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513671 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb2c72e6-372c-41c1-ba29-d47c0b521f34-serviceca\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513688 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5beb53b1-b44b-47eb-84d8-31660627d4a3-audit-log\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513701 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30634e3d-5f01-4b9b-88ad-03893a2b444e-konnectivity-ca\") pod \"konnectivity-agent-h8rzc\" (UID: \"30634e3d-5f01-4b9b-88ad-03893a2b444e\") " pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513720 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-host\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513760 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/236ec382-dbd5-4974-9454-43b39e990c77-cert\") pod \"ingress-canary-csxp6\" (UID: \"236ec382-dbd5-4974-9454-43b39e990c77\") " pod="openshift-ingress-canary/ingress-canary-csxp6" Oct 30 16:20:39.513809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513799 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9a1b81-45d0-46a0-a771-84d88046f28e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513826 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513854 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513881 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd2b0c8e-8194-456d-8299-c43b1688404a-tmp-dir\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513958 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd2b0c8e-8194-456d-8299-c43b1688404a-tmp-dir\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513988 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9ca822f3-98ef-487f-a89d-1a1004beeda1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.513998 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-registration-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514114 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9a1b81-45d0-46a0-a771-84d88046f28e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514145 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81a6d8ed-431a-4b9b-bd73-d13b01208249-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514145 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-system-cni-dir\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514208 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/236ec382-dbd5-4974-9454-43b39e990c77-cert\") pod \"ingress-canary-csxp6\" (UID: \"236ec382-dbd5-4974-9454-43b39e990c77\") " pod="openshift-ingress-canary/ingress-canary-csxp6" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514212 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b36b8f12-981e-46a8-b422-632babc5d4a7-original-pull-secret\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514214 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514259 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-systemd-units\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514284 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-log-socket\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514322 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514336 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb2c72e6-372c-41c1-ba29-d47c0b521f34-serviceca\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514390 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblkc\" (UniqueName: \"kubernetes.io/projected/cd2b0c8e-8194-456d-8299-c43b1688404a-kube-api-access-cblkc\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514423 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovnkube-script-lib\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514453 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-socket-dir-parent\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514482 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-cni-multus\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514509 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-textfile\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514532 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-ovn\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514540 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514565 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8zmd\" (UniqueName: \"kubernetes.io/projected/791cb17f-1c21-414c-9f90-196b26aa3570-kube-api-access-p8zmd\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514585 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-secret-metrics-server-client-certs\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514624 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-textfile\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514399 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.514858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514770 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-ovnkube-script-lib\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.515629 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.514883 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5beb53b1-b44b-47eb-84d8-31660627d4a3-secret-metrics-server-client-certs\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.530809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.530783 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jd74\" (UniqueName: \"kubernetes.io/projected/3e07b376-b6a8-4d05-b6f4-705695c0e92b-kube-api-access-2jd74\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:20:39.531419 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.531396 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rbfc\" (UniqueName: \"kubernetes.io/projected/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-kube-api-access-2rbfc\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.531707 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.531686 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57d5q\" (UniqueName: \"kubernetes.io/projected/67036dae-0e4a-473b-af97-e0ee20d824b8-kube-api-access-57d5q\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.531905 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.531890 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tn95\" (UniqueName: \"kubernetes.io/projected/be9a1b81-45d0-46a0-a771-84d88046f28e-kube-api-access-6tn95\") pod \"kube-state-metrics-5dc8d7d89b-5l2t5\" (UID: \"be9a1b81-45d0-46a0-a771-84d88046f28e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5l2t5" Oct 30 16:20:39.532163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.532142 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwhlq\" (UniqueName: \"kubernetes.io/projected/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-kube-api-access-lwhlq\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.532454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.532436 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6c2l\" (UniqueName: \"kubernetes.io/projected/ddebdc11-43b3-4787-a3ff-bf95150f8d27-kube-api-access-s6c2l\") pod \"network-metrics-daemon-4856h\" (UID: \"ddebdc11-43b3-4787-a3ff-bf95150f8d27\") " pod="openshift-multus/network-metrics-daemon-4856h" Oct 30 16:20:39.532508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.532491 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7bv\" (UniqueName: \"kubernetes.io/projected/3d0efab9-17c2-4f08-9433-02670766f69c-kube-api-access-4n7bv\") pod \"openshift-state-metrics-7bd7489957-pdk47\" (UID: \"3d0efab9-17c2-4f08-9433-02670766f69c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pdk47" Oct 30 16:20:39.532563 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.532546 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99d2h\" (UniqueName: \"kubernetes.io/projected/fb2c72e6-372c-41c1-ba29-d47c0b521f34-kube-api-access-99d2h\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:20:39.532613 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.532549 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdx66\" (UniqueName: \"kubernetes.io/projected/9ca822f3-98ef-487f-a89d-1a1004beeda1-kube-api-access-fdx66\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:39.533212 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.533191 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkz5f\" (UniqueName: \"kubernetes.io/projected/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-kube-api-access-qkz5f\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.533397 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.533382 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk6g4\" (UniqueName: \"kubernetes.io/projected/81a6d8ed-431a-4b9b-bd73-d13b01208249-kube-api-access-bk6g4\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.533895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.533873 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skbh\" (UniqueName: \"kubernetes.io/projected/d90a91c8-f7a6-4526-a13c-ce484713b6b5-kube-api-access-8skbh\") pod \"network-check-target-mv4lc\" (UID: \"d90a91c8-f7a6-4526-a13c-ce484713b6b5\") " pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:20:39.534342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.534321 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qnf9\" (UniqueName: \"kubernetes.io/projected/a2196112-4441-49a3-b48f-1b80bf426e29-kube-api-access-4qnf9\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.534389 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.534358 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln9pm\" (UniqueName: \"kubernetes.io/projected/236ec382-dbd5-4974-9454-43b39e990c77-kube-api-access-ln9pm\") pod \"ingress-canary-csxp6\" (UID: \"236ec382-dbd5-4974-9454-43b39e990c77\") " pod="openshift-ingress-canary/ingress-canary-csxp6" Oct 30 16:20:39.534573 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.534557 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblkc\" (UniqueName: \"kubernetes.io/projected/cd2b0c8e-8194-456d-8299-c43b1688404a-kube-api-access-cblkc\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:20:39.537144 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.537124 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8zmd\" (UniqueName: \"kubernetes.io/projected/791cb17f-1c21-414c-9f90-196b26aa3570-kube-api-access-p8zmd\") pod \"dns-default-vp4m7\" (UID: \"791cb17f-1c21-414c-9f90-196b26aa3570\") " pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:39.539402 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.539384 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpkkc\" (UniqueName: \"kubernetes.io/projected/5beb53b1-b44b-47eb-84d8-31660627d4a3-kube-api-access-xpkkc\") pod \"metrics-server-5d79986586-bccmm\" (UID: \"5beb53b1-b44b-47eb-84d8-31660627d4a3\") " pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:39.615841 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615758 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.615841 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615800 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-root\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.615841 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615822 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-run\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.615841 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615844 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-device-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615867 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-cni-netd\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615886 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-root\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615889 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615892 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-wtmp\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615918 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-device-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615922 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-cni-netd\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615950 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-sys\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615918 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-run\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.615994 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-var-lib-kubelet\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616014 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-node-exporter-wtmp\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616026 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-sys\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616016 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-var-lib-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616066 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-cni-bin\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616082 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-netns\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616091 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-var-lib-kubelet\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616099 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-lib-modules\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616119 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e07b376-b6a8-4d05-b6f4-705695c0e92b-host-slash\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616123 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-cni-bin\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616118 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-var-lib-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616132 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-sys\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616156 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae1acb11-b25d-43d5-b027-5edcf8e5fd1b-sys\") pod \"node-exporter-b9fbq\" (UID: \"ae1acb11-b25d-43d5-b027-5edcf8e5fd1b\") " pod="openshift-monitoring/node-exporter-b9fbq" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616161 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-netns\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616178 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e07b376-b6a8-4d05-b6f4-705695c0e92b-host-slash\") pod \"iptables-alerter-c2tkd\" (UID: \"3e07b376-b6a8-4d05-b6f4-705695c0e92b\") " pod="openshift-network-operator/iptables-alerter-c2tkd" Oct 30 16:20:39.616178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616162 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-k8s-cni-cncf-io\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616207 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-cni-bin\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616189 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-k8s-cni-cncf-io\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616227 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysctl-d\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616231 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-lib-modules\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616247 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-cni-bin\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616249 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-kubernetes\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616275 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-kubernetes\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616279 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-run-netns\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616295 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-cnibin\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616307 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysctl-d\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616311 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-systemd\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616330 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-cnibin\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616336 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-run-netns\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616342 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-systemd\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616354 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616375 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-host\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616393 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-registration-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616412 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-system-cni-dir\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616437 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-systemd-units\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616443 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-host\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616443 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-registration-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616457 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-system-cni-dir\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616473 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81a6d8ed-431a-4b9b-bd73-d13b01208249-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vcfvx\" (UID: \"81a6d8ed-431a-4b9b-bd73-d13b01208249\") " pod="openshift-multus/multus-additional-cni-plugins-vcfvx" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616479 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-systemd-units\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616477 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-log-socket\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616504 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-log-socket\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616517 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-socket-dir-parent\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616543 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-cni-multus\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616564 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-socket-dir-parent\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616568 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-ovn\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616594 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-cni-multus\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616593 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysconfig\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616619 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-systemd\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616631 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysconfig\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616651 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616667 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-ovn\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616711 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-systemd\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616725 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616739 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-system-cni-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616760 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ca822f3-98ef-487f-a89d-1a1004beeda1-crio-socket\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616774 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-hostroot\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616789 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-multus-certs\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616803 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-cni-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616820 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-hostroot\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616827 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb2c72e6-372c-41c1-ba29-d47c0b521f34-host\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616851 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-cni-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616854 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb2c72e6-372c-41c1-ba29-d47c0b521f34-host\") pod \"node-ca-46t9k\" (UID: \"fb2c72e6-372c-41c1-ba29-d47c0b521f34\") " pod="openshift-image-registry/node-ca-46t9k" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616827 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9ca822f3-98ef-487f-a89d-1a1004beeda1-crio-socket\") pod \"insights-runtime-extractor-n8h27\" (UID: \"9ca822f3-98ef-487f-a89d-1a1004beeda1\") " pod="openshift-insights/insights-runtime-extractor-n8h27" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616859 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-socket-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616886 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-etc-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616895 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-socket-dir\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616903 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-modprobe-d\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616924 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-slash\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616945 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-etc-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616948 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616860 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-run-multus-certs\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616866 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-system-cni-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.616966 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-slash\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617001 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-run-ovn-kubernetes\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617029 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b36b8f12-981e-46a8-b422-632babc5d4a7-dbus\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617036 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-modprobe-d\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617049 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-kubelet\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617069 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-cnibin\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617075 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b36b8f12-981e-46a8-b422-632babc5d4a7-dbus\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617092 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-kubelet\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617107 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-cnibin\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617110 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-kubelet\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617131 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-etc-kubernetes\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617149 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-host-var-lib-kubelet\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.617148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617189 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-node-log\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617211 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-sys-fs\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617244 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617250 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-etc-kubernetes\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617264 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2-sys-fs\") pod \"aws-ebs-csi-driver-node-2k7mv\" (UID: \"5b54ca32-bcd3-4baa-bd63-0ee35bb5d6e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2k7mv" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617266 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-node-log\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617299 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-conf-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617314 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b36b8f12-981e-46a8-b422-632babc5d4a7-kubelet-config\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617331 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd2b0c8e-8194-456d-8299-c43b1688404a-hosts-file\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617336 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67036dae-0e4a-473b-af97-e0ee20d824b8-multus-conf-dir\") pod \"multus-qdzgt\" (UID: \"67036dae-0e4a-473b-af97-e0ee20d824b8\") " pod="openshift-multus/multus-qdzgt" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617338 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b36b8f12-981e-46a8-b422-632babc5d4a7-kubelet-config\") pod \"global-pull-secret-syncer-lt2pz\" (UID: \"b36b8f12-981e-46a8-b422-632babc5d4a7\") " pod="kube-system/global-pull-secret-syncer-lt2pz" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617356 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617359 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysctl-conf\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617367 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cd2b0c8e-8194-456d-8299-c43b1688404a-hosts-file\") pod \"node-resolver-7rnq2\" (UID: \"cd2b0c8e-8194-456d-8299-c43b1688404a\") " pod="openshift-dns/node-resolver-7rnq2" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617418 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617441 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a2196112-4441-49a3-b48f-1b80bf426e29-etc-sysctl-conf\") pod \"tuned-4g45x\" (UID: \"a2196112-4441-49a3-b48f-1b80bf426e29\") " pod="openshift-cluster-node-tuning-operator/tuned-4g45x" Oct 30 16:20:39.619082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:39.617478 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b13dfe3-9da5-4be8-9051-b1bbff1b083b-run-openvswitch\") pod \"ovnkube-node-v4lzf\" (UID: \"4b13dfe3-9da5-4be8-9051-b1bbff1b083b\") " pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:40.845381 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:40.845348 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:42.016088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:42.016053 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vp4m7" Oct 30 16:20:46.159023 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:46.158964 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:20:49.030127 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:49.030097 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5d79986586-bccmm" Oct 30 16:20:49.577536 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:49.577503 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:49.577759 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:49.577733 5838 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:51.607628 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:51.607596 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h8rzc" Oct 30 16:20:54.109237 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:54.109205 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v4lzf" Oct 30 16:20:58.603929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.603897 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5f47789cbf-4q68x"] Oct 30 16:20:58.606245 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.606227 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.638732 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.638689 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 16:20:58.638732 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.638687 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 16:20:58.638732 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.638722 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 16:20:58.639017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.638687 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 16:20:58.639017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.638850 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fkbhl\"" Oct 30 16:20:58.639344 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.639324 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f47789cbf-4q68x"] Oct 30 16:20:58.687117 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.687085 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 16:20:58.698829 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.698806 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 16:20:58.739164 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.739139 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/51fcc480-c8f0-46f6-8092-04e12b809d95-telemeter-client-tls\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.739229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.739176 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51fcc480-c8f0-46f6-8092-04e12b809d95-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.739229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.739198 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51fcc480-c8f0-46f6-8092-04e12b809d95-serving-certs-ca-bundle\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.739334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.739309 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51fcc480-c8f0-46f6-8092-04e12b809d95-metrics-client-ca\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.739364 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.739351 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51fcc480-c8f0-46f6-8092-04e12b809d95-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.739421 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.739407 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/51fcc480-c8f0-46f6-8092-04e12b809d95-federate-client-tls\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.739454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.739444 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/51fcc480-c8f0-46f6-8092-04e12b809d95-secret-telemeter-client\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.739485 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.739462 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbxdg\" (UniqueName: \"kubernetes.io/projected/51fcc480-c8f0-46f6-8092-04e12b809d95-kube-api-access-dbxdg\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.840672 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.840636 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/51fcc480-c8f0-46f6-8092-04e12b809d95-federate-client-tls\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.840772 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.840684 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/51fcc480-c8f0-46f6-8092-04e12b809d95-secret-telemeter-client\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.840772 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.840703 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbxdg\" (UniqueName: \"kubernetes.io/projected/51fcc480-c8f0-46f6-8092-04e12b809d95-kube-api-access-dbxdg\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.840772 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.840732 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/51fcc480-c8f0-46f6-8092-04e12b809d95-telemeter-client-tls\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.840772 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.840755 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51fcc480-c8f0-46f6-8092-04e12b809d95-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.840940 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.840775 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51fcc480-c8f0-46f6-8092-04e12b809d95-serving-certs-ca-bundle\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.840940 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.840804 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51fcc480-c8f0-46f6-8092-04e12b809d95-metrics-client-ca\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.840940 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.840836 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51fcc480-c8f0-46f6-8092-04e12b809d95-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.841152 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.841126 5838 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:20:58.841656 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.841620 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51fcc480-c8f0-46f6-8092-04e12b809d95-serving-certs-ca-bundle\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.841770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.841662 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51fcc480-c8f0-46f6-8092-04e12b809d95-metrics-client-ca\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.841986 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.841953 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51fcc480-c8f0-46f6-8092-04e12b809d95-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.843541 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.843519 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/51fcc480-c8f0-46f6-8092-04e12b809d95-secret-telemeter-client\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.843715 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.843686 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/51fcc480-c8f0-46f6-8092-04e12b809d95-telemeter-client-tls\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.843961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.843944 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/51fcc480-c8f0-46f6-8092-04e12b809d95-federate-client-tls\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.844055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.844025 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51fcc480-c8f0-46f6-8092-04e12b809d95-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.868444 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.868389 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbxdg\" (UniqueName: \"kubernetes.io/projected/51fcc480-c8f0-46f6-8092-04e12b809d95-kube-api-access-dbxdg\") pod \"telemeter-client-5f47789cbf-4q68x\" (UID: \"51fcc480-c8f0-46f6-8092-04e12b809d95\") " pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:58.920264 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:58.920237 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" Oct 30 16:20:59.253941 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:59.253908 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f47789cbf-4q68x"] Oct 30 16:20:59.259147 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:20:59.259121 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fcc480_c8f0_46f6_8092_04e12b809d95.slice/crio-e8c336803b628152c114cbc29664de8415458c267a83862d4bfc56831fc07e60 WatchSource:0}: Error finding container e8c336803b628152c114cbc29664de8415458c267a83862d4bfc56831fc07e60: Status 404 returned error can't find the container with id e8c336803b628152c114cbc29664de8415458c267a83862d4bfc56831fc07e60 Oct 30 16:20:59.261127 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:59.261107 5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:20:59.503355 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:20:59.503322 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" event={"ID":"51fcc480-c8f0-46f6-8092-04e12b809d95","Type":"ContainerStarted","Data":"e8c336803b628152c114cbc29664de8415458c267a83862d4bfc56831fc07e60"} Oct 30 16:21:01.512504 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:21:01.512468 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" event={"ID":"51fcc480-c8f0-46f6-8092-04e12b809d95","Type":"ContainerStarted","Data":"331882af52a41c11a7607558f2dcf1a8cae93d23c414776f958e325e475d1fb6"} Oct 30 16:21:01.802013 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:21:01.801906 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:21:02.521133 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:21:02.521097 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" event={"ID":"51fcc480-c8f0-46f6-8092-04e12b809d95","Type":"ContainerStarted","Data":"1ae5a0fd72bcca758f031fb14670454bc8a1860e6493531fe09870b1758acbb3"} Oct 30 16:21:02.521623 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:21:02.521179 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" event={"ID":"51fcc480-c8f0-46f6-8092-04e12b809d95","Type":"ContainerStarted","Data":"3ec9f1b5db37c8dc4d529bc43402a01925dd34c63c882fc36405178191ca365e"} Oct 30 16:21:02.589981 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:21:02.589915 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5f47789cbf-4q68x" podStartSLOduration=2.071741113 podStartE2EDuration="4.589902694s" totalImagesPullingTime="2.518161581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:02.589789465 +0000 UTC m=+24.338378853" watchObservedRunningTime="2026-10-30 16:21:02.589902694 +0000 UTC m=+24.338492081" Oct 30 16:21:06.162024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:21:06.161991 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mv4lc" Oct 30 16:21:31.794989 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:21:31.794930 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:22:01.795994 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:22:01.795937 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:22:31.796014 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:22:31.795915 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:23:01.795227 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:23:01.795186 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:23:31.795192 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:23:31.795149 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:23:39.807021 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.806966 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn"] Oct 30 16:23:39.809698 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.809674 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:39.835271 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.835248 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:23:39.835271 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.835262 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:23:39.835271 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.835269 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-mqxt8\"" Oct 30 16:23:39.851442 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.851423 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn"] Oct 30 16:23:39.853208 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.853179 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g66m8\" (UniqueName: \"kubernetes.io/projected/f0e595aa-59f5-460d-8ffc-720c780f66e7-kube-api-access-g66m8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn\" (UID: \"f0e595aa-59f5-460d-8ffc-720c780f66e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:39.853375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.853235 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn\" (UID: \"f0e595aa-59f5-460d-8ffc-720c780f66e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:39.853441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.853393 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn\" (UID: \"f0e595aa-59f5-460d-8ffc-720c780f66e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:39.954750 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.954717 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn\" (UID: \"f0e595aa-59f5-460d-8ffc-720c780f66e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:39.954923 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.954769 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g66m8\" (UniqueName: \"kubernetes.io/projected/f0e595aa-59f5-460d-8ffc-720c780f66e7-kube-api-access-g66m8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn\" (UID: \"f0e595aa-59f5-460d-8ffc-720c780f66e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:39.954923 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.954797 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn\" (UID: \"f0e595aa-59f5-460d-8ffc-720c780f66e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:39.955177 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.955156 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn\" (UID: \"f0e595aa-59f5-460d-8ffc-720c780f66e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:39.955215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.955192 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn\" (UID: \"f0e595aa-59f5-460d-8ffc-720c780f66e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:39.978829 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:39.978798 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g66m8\" (UniqueName: \"kubernetes.io/projected/f0e595aa-59f5-460d-8ffc-720c780f66e7-kube-api-access-g66m8\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn\" (UID: \"f0e595aa-59f5-460d-8ffc-720c780f66e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:40.122619 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:40.122537 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:40.415259 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:40.415178 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn"] Oct 30 16:23:40.421030 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:23:40.421000 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e595aa_59f5_460d_8ffc_720c780f66e7.slice/crio-6871333081f5ab9a58bfefba51dd4d7cd4ebd070c4ef0e7334ac33ebbcbdea51 WatchSource:0}: Error finding container 6871333081f5ab9a58bfefba51dd4d7cd4ebd070c4ef0e7334ac33ebbcbdea51: Status 404 returned error can't find the container with id 6871333081f5ab9a58bfefba51dd4d7cd4ebd070c4ef0e7334ac33ebbcbdea51 Oct 30 16:23:41.011397 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:41.011362 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" event={"ID":"f0e595aa-59f5-460d-8ffc-720c780f66e7","Type":"ContainerStarted","Data":"6871333081f5ab9a58bfefba51dd4d7cd4ebd070c4ef0e7334ac33ebbcbdea51"} Oct 30 16:23:46.032628 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:46.032595 5838 generic.go:356] "Generic (PLEG): container finished" podID="f0e595aa-59f5-460d-8ffc-720c780f66e7" containerID="6d8a546cc1ed99879ebaf1a9518a0abd4bf663a6285f9a11c722cdc3e6f5729c" exitCode=0 Oct 30 16:23:46.033033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:46.032645 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" event={"ID":"f0e595aa-59f5-460d-8ffc-720c780f66e7","Type":"ContainerDied","Data":"6d8a546cc1ed99879ebaf1a9518a0abd4bf663a6285f9a11c722cdc3e6f5729c"} Oct 30 16:23:50.049527 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:50.049441 5838 generic.go:356] "Generic (PLEG): container finished" podID="f0e595aa-59f5-460d-8ffc-720c780f66e7" containerID="189b86b09f72305c1f2ef1fa2237410b8b22916ac6b2224ac9cba07caaea8a43" exitCode=0 Oct 30 16:23:50.049527 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:50.049486 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" event={"ID":"f0e595aa-59f5-460d-8ffc-720c780f66e7","Type":"ContainerDied","Data":"189b86b09f72305c1f2ef1fa2237410b8b22916ac6b2224ac9cba07caaea8a43"} Oct 30 16:23:56.073542 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:56.073500 5838 generic.go:356] "Generic (PLEG): container finished" podID="f0e595aa-59f5-460d-8ffc-720c780f66e7" containerID="d2091e3dc14fc53765049b2ba83f235e9dfec580395168d603e1e69dcbe89dd7" exitCode=0 Oct 30 16:23:56.073931 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:56.073563 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" event={"ID":"f0e595aa-59f5-460d-8ffc-720c780f66e7","Type":"ContainerDied","Data":"d2091e3dc14fc53765049b2ba83f235e9dfec580395168d603e1e69dcbe89dd7"} Oct 30 16:23:57.199822 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:57.199798 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:57.291182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:57.291149 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-util\" (UniqueName: \"kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-util\") pod \"f0e595aa-59f5-460d-8ffc-720c780f66e7\" (UID: \"f0e595aa-59f5-460d-8ffc-720c780f66e7\") " Oct 30 16:23:57.291317 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:57.291228 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-bundle\") pod \"f0e595aa-59f5-460d-8ffc-720c780f66e7\" (UID: \"f0e595aa-59f5-460d-8ffc-720c780f66e7\") " Oct 30 16:23:57.291317 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:57.291264 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e595aa-59f5-460d-8ffc-720c780f66e7-kube-api-access-g66m8\" (UniqueName: \"kubernetes.io/projected/f0e595aa-59f5-460d-8ffc-720c780f66e7-kube-api-access-g66m8\") pod \"f0e595aa-59f5-460d-8ffc-720c780f66e7\" (UID: \"f0e595aa-59f5-460d-8ffc-720c780f66e7\") " Oct 30 16:23:57.291832 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:57.291806 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-bundle" pod "f0e595aa-59f5-460d-8ffc-720c780f66e7" (UID: "f0e595aa-59f5-460d-8ffc-720c780f66e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:23:57.293465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:57.293433 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e595aa-59f5-460d-8ffc-720c780f66e7-kube-api-access-g66m8" pod "f0e595aa-59f5-460d-8ffc-720c780f66e7" (UID: "f0e595aa-59f5-460d-8ffc-720c780f66e7"). InnerVolumeSpecName "kube-api-access-g66m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:23:57.294920 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:57.294894 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-util" pod "f0e595aa-59f5-460d-8ffc-720c780f66e7" (UID: "f0e595aa-59f5-460d-8ffc-720c780f66e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:23:57.391728 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:57.391639 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g66m8\" (UniqueName: \"kubernetes.io/projected/f0e595aa-59f5-460d-8ffc-720c780f66e7-kube-api-access-g66m8\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:23:57.391728 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:57.391670 5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:23:57.391728 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:57.391683 5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0e595aa-59f5-460d-8ffc-720c780f66e7-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:23:58.082353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:58.082314 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" event={"ID":"f0e595aa-59f5-460d-8ffc-720c780f66e7","Type":"ContainerDied","Data":"6871333081f5ab9a58bfefba51dd4d7cd4ebd070c4ef0e7334ac33ebbcbdea51"} Oct 30 16:23:58.082524 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:58.082357 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651clh4nn" Oct 30 16:23:58.082524 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:23:58.082390 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6871333081f5ab9a58bfefba51dd4d7cd4ebd070c4ef0e7334ac33ebbcbdea51" Oct 30 16:24:01.795931 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:24:01.795889 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:24:03.025936 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.025895 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt"] Oct 30 16:24:03.026450 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.026292 5838 cpu_manager.go:414] "containerMap: removing container" podUID="f0e595aa-59f5-460d-8ffc-720c780f66e7" containerName="pull" Oct 30 16:24:03.026450 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.026311 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e595aa-59f5-460d-8ffc-720c780f66e7" containerName="pull" Oct 30 16:24:03.026450 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.026335 5838 cpu_manager.go:414] "containerMap: removing container" podUID="f0e595aa-59f5-460d-8ffc-720c780f66e7" containerName="util" Oct 30 16:24:03.026450 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.026343 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e595aa-59f5-460d-8ffc-720c780f66e7" containerName="util" Oct 30 16:24:03.026450 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.026352 5838 cpu_manager.go:414] "containerMap: removing container" podUID="f0e595aa-59f5-460d-8ffc-720c780f66e7" containerName="extract" Oct 30 16:24:03.026450 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.026358 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e595aa-59f5-460d-8ffc-720c780f66e7" containerName="extract" Oct 30 16:24:03.026450 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.026412 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e595aa-59f5-460d-8ffc-720c780f66e7" containerName="extract" Oct 30 16:24:03.062346 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.062312 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt"] Oct 30 16:24:03.062531 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.062465 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt" Oct 30 16:24:03.076904 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.076867 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 16:24:03.077118 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.076965 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:24:03.077118 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.077082 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-79ggf\"" Oct 30 16:24:03.132891 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.132851 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8988f7d0-ae91-42da-8f57-35e7c7242c0b-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-9h9wt\" (UID: \"8988f7d0-ae91-42da-8f57-35e7c7242c0b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt" Oct 30 16:24:03.133090 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.132921 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswnp\" (UniqueName: \"kubernetes.io/projected/8988f7d0-ae91-42da-8f57-35e7c7242c0b-kube-api-access-bswnp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-9h9wt\" (UID: \"8988f7d0-ae91-42da-8f57-35e7c7242c0b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt" Oct 30 16:24:03.234303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.234264 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswnp\" (UniqueName: \"kubernetes.io/projected/8988f7d0-ae91-42da-8f57-35e7c7242c0b-kube-api-access-bswnp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-9h9wt\" (UID: \"8988f7d0-ae91-42da-8f57-35e7c7242c0b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt" Oct 30 16:24:03.234491 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.234321 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8988f7d0-ae91-42da-8f57-35e7c7242c0b-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-9h9wt\" (UID: \"8988f7d0-ae91-42da-8f57-35e7c7242c0b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt" Oct 30 16:24:03.234664 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.234648 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8988f7d0-ae91-42da-8f57-35e7c7242c0b-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-9h9wt\" (UID: \"8988f7d0-ae91-42da-8f57-35e7c7242c0b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt" Oct 30 16:24:03.258667 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.258631 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswnp\" (UniqueName: \"kubernetes.io/projected/8988f7d0-ae91-42da-8f57-35e7c7242c0b-kube-api-access-bswnp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-9h9wt\" (UID: \"8988f7d0-ae91-42da-8f57-35e7c7242c0b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt" Oct 30 16:24:03.384784 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.384694 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt" Oct 30 16:24:03.727802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:03.727712 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt"] Oct 30 16:24:03.747467 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:24:03.747430 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8988f7d0_ae91_42da_8f57_35e7c7242c0b.slice/crio-c2f7d05bab1a4c3b119915dc7545196fa96e632a527066c6b6a178b67d2d8fae WatchSource:0}: Error finding container c2f7d05bab1a4c3b119915dc7545196fa96e632a527066c6b6a178b67d2d8fae: Status 404 returned error can't find the container with id c2f7d05bab1a4c3b119915dc7545196fa96e632a527066c6b6a178b67d2d8fae Oct 30 16:24:04.104888 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:04.104847 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt" event={"ID":"8988f7d0-ae91-42da-8f57-35e7c7242c0b","Type":"ContainerStarted","Data":"c2f7d05bab1a4c3b119915dc7545196fa96e632a527066c6b6a178b67d2d8fae"} Oct 30 16:24:06.114097 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:06.114055 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt" event={"ID":"8988f7d0-ae91-42da-8f57-35e7c7242c0b","Type":"ContainerStarted","Data":"5e4a262c191d5d6abc9047007b3e0827639ccec8fc6041ff7c4d8b0f5e7f94ba"} Oct 30 16:24:06.181618 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:06.181540 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9h9wt" podStartSLOduration=2.283332349 podStartE2EDuration="4.181521885s" totalImagesPullingTime="1.898189536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:06.169577638 +0000 UTC m=+207.918167027" watchObservedRunningTime="2026-10-30 16:24:06.181521885 +0000 UTC m=+207.930111273" Oct 30 16:24:07.485989 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.485937 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm"] Oct 30 16:24:07.502050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.502021 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:07.510146 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.510119 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm"] Oct 30 16:24:07.521201 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.521175 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:24:07.521360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.521216 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-jf6cj\"" Oct 30 16:24:07.521360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.521265 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:24:07.570878 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.570843 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm\" (UID: \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:07.571039 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.570887 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n489\" (UniqueName: \"kubernetes.io/projected/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-kube-api-access-7n489\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm\" (UID: \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:07.571039 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.570984 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm\" (UID: \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:07.672377 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.672338 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm\" (UID: \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:07.672555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.672388 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n489\" (UniqueName: \"kubernetes.io/projected/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-kube-api-access-7n489\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm\" (UID: \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:07.672555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.672421 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm\" (UID: \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:07.672833 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.672735 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm\" (UID: \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:07.672833 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.672761 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm\" (UID: \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:07.696932 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.696908 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n489\" (UniqueName: \"kubernetes.io/projected/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-kube-api-access-7n489\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm\" (UID: \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:07.850720 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:07.850643 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:08.156801 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:08.156723 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm"] Oct 30 16:24:08.162093 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:24:08.162061 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd9ed80_fe6d_4cd6_96fa_dc2af87c4a66.slice/crio-f21ffea4e2ce151c77aebba2dc566a489511a52563d47dd67e227dac1f06048a WatchSource:0}: Error finding container f21ffea4e2ce151c77aebba2dc566a489511a52563d47dd67e227dac1f06048a: Status 404 returned error can't find the container with id f21ffea4e2ce151c77aebba2dc566a489511a52563d47dd67e227dac1f06048a Oct 30 16:24:09.126395 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:09.126356 5838 generic.go:356] "Generic (PLEG): container finished" podID="1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" containerID="68fd658423377ba2eea3bc0c2215fc2d3d901be480be445d89ee1c00f95d9920" exitCode=0 Oct 30 16:24:09.126797 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:09.126422 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" event={"ID":"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66","Type":"ContainerDied","Data":"68fd658423377ba2eea3bc0c2215fc2d3d901be480be445d89ee1c00f95d9920"} Oct 30 16:24:09.126797 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:09.126464 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" event={"ID":"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66","Type":"ContainerStarted","Data":"f21ffea4e2ce151c77aebba2dc566a489511a52563d47dd67e227dac1f06048a"} Oct 30 16:24:11.136090 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:11.136057 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" event={"ID":"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66","Type":"ContainerStarted","Data":"8aaa662bc477cf71e97011489ee923047817c0c18cac8cb5361d1a1f9bd93d13"} Oct 30 16:24:11.770800 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:11.770769 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb"] Oct 30 16:24:11.772944 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:11.772920 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb" Oct 30 16:24:11.787315 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:11.787271 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 16:24:11.787315 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:11.787282 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 16:24:11.787491 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:11.787344 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-pft8v\"" Oct 30 16:24:11.793346 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:11.793325 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb"] Oct 30 16:24:11.909112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:11.909073 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ea30dd9-07c8-4815-a27f-95265f501b00-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-g6rrb\" (UID: \"2ea30dd9-07c8-4815-a27f-95265f501b00\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb" Oct 30 16:24:11.909318 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:11.909129 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh9zq\" (UniqueName: \"kubernetes.io/projected/2ea30dd9-07c8-4815-a27f-95265f501b00-kube-api-access-rh9zq\") pod \"cert-manager-cainjector-5c57bbd4fc-g6rrb\" (UID: \"2ea30dd9-07c8-4815-a27f-95265f501b00\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb" Oct 30 16:24:12.010264 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:12.010226 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ea30dd9-07c8-4815-a27f-95265f501b00-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-g6rrb\" (UID: \"2ea30dd9-07c8-4815-a27f-95265f501b00\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb" Oct 30 16:24:12.010443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:12.010279 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh9zq\" (UniqueName: \"kubernetes.io/projected/2ea30dd9-07c8-4815-a27f-95265f501b00-kube-api-access-rh9zq\") pod \"cert-manager-cainjector-5c57bbd4fc-g6rrb\" (UID: \"2ea30dd9-07c8-4815-a27f-95265f501b00\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb" Oct 30 16:24:12.033367 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:12.033293 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ea30dd9-07c8-4815-a27f-95265f501b00-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-g6rrb\" (UID: \"2ea30dd9-07c8-4815-a27f-95265f501b00\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb" Oct 30 16:24:12.033491 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:12.033453 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh9zq\" (UniqueName: \"kubernetes.io/projected/2ea30dd9-07c8-4815-a27f-95265f501b00-kube-api-access-rh9zq\") pod \"cert-manager-cainjector-5c57bbd4fc-g6rrb\" (UID: \"2ea30dd9-07c8-4815-a27f-95265f501b00\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb" Oct 30 16:24:12.093004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:12.092947 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb" Oct 30 16:24:12.143843 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:12.143804 5838 generic.go:356] "Generic (PLEG): container finished" podID="1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" containerID="8aaa662bc477cf71e97011489ee923047817c0c18cac8cb5361d1a1f9bd93d13" exitCode=0 Oct 30 16:24:12.144615 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:12.143929 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" event={"ID":"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66","Type":"ContainerDied","Data":"8aaa662bc477cf71e97011489ee923047817c0c18cac8cb5361d1a1f9bd93d13"} Oct 30 16:24:12.393679 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:12.393577 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb"] Oct 30 16:24:12.396385 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:24:12.396358 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ea30dd9_07c8_4815_a27f_95265f501b00.slice/crio-1928deba980d9c13d6e03274d9fdc679aa81fea988a991f1ae02d8cadbd52506 WatchSource:0}: Error finding container 1928deba980d9c13d6e03274d9fdc679aa81fea988a991f1ae02d8cadbd52506: Status 404 returned error can't find the container with id 1928deba980d9c13d6e03274d9fdc679aa81fea988a991f1ae02d8cadbd52506 Oct 30 16:24:13.149897 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:13.149860 5838 generic.go:356] "Generic (PLEG): container finished" podID="1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" containerID="24470601c276ad0ecaa29117aa7a5a9ecbf3c14c6a7d25bd60f39a6d3d3b95f2" exitCode=0 Oct 30 16:24:13.150360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:13.149951 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" event={"ID":"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66","Type":"ContainerDied","Data":"24470601c276ad0ecaa29117aa7a5a9ecbf3c14c6a7d25bd60f39a6d3d3b95f2"} Oct 30 16:24:13.151144 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:13.151123 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb" event={"ID":"2ea30dd9-07c8-4815-a27f-95265f501b00","Type":"ContainerStarted","Data":"1928deba980d9c13d6e03274d9fdc679aa81fea988a991f1ae02d8cadbd52506"} Oct 30 16:24:14.094931 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.094891 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb"] Oct 30 16:24:14.097770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.097744 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" Oct 30 16:24:14.125072 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.125039 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb"] Oct 30 16:24:14.125240 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.125140 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-brlnq\"" Oct 30 16:24:14.229552 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.229508 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f6b4090-8bc3-4396-9f43-9150857119f8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5wtjb\" (UID: \"9f6b4090-8bc3-4396-9f43-9150857119f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" Oct 30 16:24:14.229909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.229571 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbf4\" (UniqueName: \"kubernetes.io/projected/9f6b4090-8bc3-4396-9f43-9150857119f8-kube-api-access-bjbf4\") pod \"cert-manager-webhook-5dbbbf75cc-5wtjb\" (UID: \"9f6b4090-8bc3-4396-9f43-9150857119f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" Oct 30 16:24:14.296705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.296681 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:14.330937 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.330892 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f6b4090-8bc3-4396-9f43-9150857119f8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5wtjb\" (UID: \"9f6b4090-8bc3-4396-9f43-9150857119f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" Oct 30 16:24:14.331132 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.330958 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbf4\" (UniqueName: \"kubernetes.io/projected/9f6b4090-8bc3-4396-9f43-9150857119f8-kube-api-access-bjbf4\") pod \"cert-manager-webhook-5dbbbf75cc-5wtjb\" (UID: \"9f6b4090-8bc3-4396-9f43-9150857119f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" Oct 30 16:24:14.353840 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.353738 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f6b4090-8bc3-4396-9f43-9150857119f8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5wtjb\" (UID: \"9f6b4090-8bc3-4396-9f43-9150857119f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" Oct 30 16:24:14.353840 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.353795 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbf4\" (UniqueName: \"kubernetes.io/projected/9f6b4090-8bc3-4396-9f43-9150857119f8-kube-api-access-bjbf4\") pod \"cert-manager-webhook-5dbbbf75cc-5wtjb\" (UID: \"9f6b4090-8bc3-4396-9f43-9150857119f8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" Oct 30 16:24:14.411140 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.411106 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" Oct 30 16:24:14.431521 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.431485 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-util\" (UniqueName: \"kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-util\") pod \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\" (UID: \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\") " Oct 30 16:24:14.431660 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.431547 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-kube-api-access-7n489\" (UniqueName: \"kubernetes.io/projected/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-kube-api-access-7n489\") pod \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\" (UID: \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\") " Oct 30 16:24:14.431660 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.431605 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-bundle\") pod \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\" (UID: \"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66\") " Oct 30 16:24:14.432072 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.432046 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-bundle" pod "1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" (UID: "1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:24:14.434051 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.434027 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-kube-api-access-7n489" pod "1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" (UID: "1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66"). InnerVolumeSpecName "kube-api-access-7n489". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:24:14.440997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.437779 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-util" pod "1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" (UID: "1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:24:14.533099 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.533049 5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:24:14.533099 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.533086 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n489\" (UniqueName: \"kubernetes.io/projected/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-kube-api-access-7n489\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:24:14.533099 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.533103 5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:24:14.710518 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:14.710445 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb"] Oct 30 16:24:14.716151 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:24:14.716119 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f6b4090_8bc3_4396_9f43_9150857119f8.slice/crio-3bcfefb481fea6a239575a9782cab5de2b7051a0e015565e901c2a8a4e1f7a25 WatchSource:0}: Error finding container 3bcfefb481fea6a239575a9782cab5de2b7051a0e015565e901c2a8a4e1f7a25: Status 404 returned error can't find the container with id 3bcfefb481fea6a239575a9782cab5de2b7051a0e015565e901c2a8a4e1f7a25 Oct 30 16:24:15.162208 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:15.162163 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" event={"ID":"1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66","Type":"ContainerDied","Data":"f21ffea4e2ce151c77aebba2dc566a489511a52563d47dd67e227dac1f06048a"} Oct 30 16:24:15.162402 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:15.162229 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f21ffea4e2ce151c77aebba2dc566a489511a52563d47dd67e227dac1f06048a" Oct 30 16:24:15.162402 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:15.162179 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85dljpm" Oct 30 16:24:15.163656 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:15.163630 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" event={"ID":"9f6b4090-8bc3-4396-9f43-9150857119f8","Type":"ContainerStarted","Data":"3bcfefb481fea6a239575a9782cab5de2b7051a0e015565e901c2a8a4e1f7a25"} Oct 30 16:24:16.168500 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:16.168463 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" event={"ID":"9f6b4090-8bc3-4396-9f43-9150857119f8","Type":"ContainerStarted","Data":"f1361e9459285439baf5b6190eccce7200d9771ba10a55a6d8331e2260f5d59d"} Oct 30 16:24:16.168984 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:16.168514 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" Oct 30 16:24:16.170011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:16.169987 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb" event={"ID":"2ea30dd9-07c8-4815-a27f-95265f501b00","Type":"ContainerStarted","Data":"6704aa9877774ae3cc4bd13e3eea015fa0960dd03ccc5c3dc50ff17f8865bac2"} Oct 30 16:24:16.208099 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:16.208050 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" podStartSLOduration=1.557968555 podStartE2EDuration="2.208036883s" totalImagesPullingTime="650.068328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:16.207487006 +0000 UTC m=+217.956076444" watchObservedRunningTime="2026-10-30 16:24:16.208036883 +0000 UTC m=+217.956626267" Oct 30 16:24:16.248759 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:16.248711 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-g6rrb" podStartSLOduration=2.386732832 podStartE2EDuration="5.248699084s" totalImagesPullingTime="2.861966252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:16.248313813 +0000 UTC m=+217.996903202" watchObservedRunningTime="2026-10-30 16:24:16.248699084 +0000 UTC m=+217.997288470" Oct 30 16:24:17.770422 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.770391 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-mlmqm"] Oct 30 16:24:17.770783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.770638 5838 cpu_manager.go:414] "containerMap: removing container" podUID="1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" containerName="util" Oct 30 16:24:17.770783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.770650 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" containerName="util" Oct 30 16:24:17.770783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.770667 5838 cpu_manager.go:414] "containerMap: removing container" podUID="1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" containerName="pull" Oct 30 16:24:17.770783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.770672 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" containerName="pull" Oct 30 16:24:17.770783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.770679 5838 cpu_manager.go:414] "containerMap: removing container" podUID="1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" containerName="extract" Oct 30 16:24:17.770783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.770685 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" containerName="extract" Oct 30 16:24:17.770783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.770736 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd9ed80-fe6d-4cd6-96fa-dc2af87c4a66" containerName="extract" Oct 30 16:24:17.772725 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.772698 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-mlmqm" Oct 30 16:24:17.797958 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.797929 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4k84c\"" Oct 30 16:24:17.798938 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.798914 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-mlmqm"] Oct 30 16:24:17.865574 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.865540 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49471d2c-2cd7-4a29-b593-d96a4e3a2829-bound-sa-token\") pod \"cert-manager-597b95cb84-mlmqm\" (UID: \"49471d2c-2cd7-4a29-b593-d96a4e3a2829\") " pod="cert-manager/cert-manager-597b95cb84-mlmqm" Oct 30 16:24:17.865753 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.865664 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84w8r\" (UniqueName: \"kubernetes.io/projected/49471d2c-2cd7-4a29-b593-d96a4e3a2829-kube-api-access-84w8r\") pod \"cert-manager-597b95cb84-mlmqm\" (UID: \"49471d2c-2cd7-4a29-b593-d96a4e3a2829\") " pod="cert-manager/cert-manager-597b95cb84-mlmqm" Oct 30 16:24:17.966909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.966868 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84w8r\" (UniqueName: \"kubernetes.io/projected/49471d2c-2cd7-4a29-b593-d96a4e3a2829-kube-api-access-84w8r\") pod \"cert-manager-597b95cb84-mlmqm\" (UID: \"49471d2c-2cd7-4a29-b593-d96a4e3a2829\") " pod="cert-manager/cert-manager-597b95cb84-mlmqm" Oct 30 16:24:17.967077 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.966917 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49471d2c-2cd7-4a29-b593-d96a4e3a2829-bound-sa-token\") pod \"cert-manager-597b95cb84-mlmqm\" (UID: \"49471d2c-2cd7-4a29-b593-d96a4e3a2829\") " pod="cert-manager/cert-manager-597b95cb84-mlmqm" Oct 30 16:24:17.989704 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.989678 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49471d2c-2cd7-4a29-b593-d96a4e3a2829-bound-sa-token\") pod \"cert-manager-597b95cb84-mlmqm\" (UID: \"49471d2c-2cd7-4a29-b593-d96a4e3a2829\") " pod="cert-manager/cert-manager-597b95cb84-mlmqm" Oct 30 16:24:17.989826 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:17.989793 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84w8r\" (UniqueName: \"kubernetes.io/projected/49471d2c-2cd7-4a29-b593-d96a4e3a2829-kube-api-access-84w8r\") pod \"cert-manager-597b95cb84-mlmqm\" (UID: \"49471d2c-2cd7-4a29-b593-d96a4e3a2829\") " pod="cert-manager/cert-manager-597b95cb84-mlmqm" Oct 30 16:24:18.082653 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:18.082577 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-mlmqm" Oct 30 16:24:18.380168 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:18.379531 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-mlmqm"] Oct 30 16:24:18.386483 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:24:18.386453 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49471d2c_2cd7_4a29_b593_d96a4e3a2829.slice/crio-3441977baf13b914e26df07c03e88c426d169e2ec1a7e67c600586ba38fb2e97 WatchSource:0}: Error finding container 3441977baf13b914e26df07c03e88c426d169e2ec1a7e67c600586ba38fb2e97: Status 404 returned error can't find the container with id 3441977baf13b914e26df07c03e88c426d169e2ec1a7e67c600586ba38fb2e97 Oct 30 16:24:19.181870 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:19.181834 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-mlmqm" event={"ID":"49471d2c-2cd7-4a29-b593-d96a4e3a2829","Type":"ContainerStarted","Data":"44038f2f4c42717537faff8f1869f94de4987fc83b6ca6cf0ea87c9e17fecda3"} Oct 30 16:24:19.182273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:19.181883 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-mlmqm" event={"ID":"49471d2c-2cd7-4a29-b593-d96a4e3a2829","Type":"ContainerStarted","Data":"3441977baf13b914e26df07c03e88c426d169e2ec1a7e67c600586ba38fb2e97"} Oct 30 16:24:19.220466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:19.220414 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-mlmqm" podStartSLOduration=2.220401128 podStartE2EDuration="2.220401128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:19.219913977 +0000 UTC m=+220.968503366" watchObservedRunningTime="2026-10-30 16:24:19.220401128 +0000 UTC m=+220.968990511" Oct 30 16:24:22.176364 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:22.176331 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" Oct 30 16:24:25.483058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.483024 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb"] Oct 30 16:24:25.485372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.485351 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb" Oct 30 16:24:25.511076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.511050 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:24:25.511219 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.511152 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 16:24:25.511360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.511336 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-mcmtf\"" Oct 30 16:24:25.512109 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.512087 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb"] Oct 30 16:24:25.626143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.626103 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsbfz\" (UniqueName: \"kubernetes.io/projected/80c7928f-4dc7-49fc-b98d-1c8d35f06b98-kube-api-access-gsbfz\") pod \"openshift-lws-operator-8465994787-l2bpb\" (UID: \"80c7928f-4dc7-49fc-b98d-1c8d35f06b98\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb" Oct 30 16:24:25.626320 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.626159 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c7928f-4dc7-49fc-b98d-1c8d35f06b98-tmp\") pod \"openshift-lws-operator-8465994787-l2bpb\" (UID: \"80c7928f-4dc7-49fc-b98d-1c8d35f06b98\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb" Oct 30 16:24:25.726527 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.726491 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c7928f-4dc7-49fc-b98d-1c8d35f06b98-tmp\") pod \"openshift-lws-operator-8465994787-l2bpb\" (UID: \"80c7928f-4dc7-49fc-b98d-1c8d35f06b98\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb" Oct 30 16:24:25.726605 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.726554 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsbfz\" (UniqueName: \"kubernetes.io/projected/80c7928f-4dc7-49fc-b98d-1c8d35f06b98-kube-api-access-gsbfz\") pod \"openshift-lws-operator-8465994787-l2bpb\" (UID: \"80c7928f-4dc7-49fc-b98d-1c8d35f06b98\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb" Oct 30 16:24:25.726876 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.726856 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80c7928f-4dc7-49fc-b98d-1c8d35f06b98-tmp\") pod \"openshift-lws-operator-8465994787-l2bpb\" (UID: \"80c7928f-4dc7-49fc-b98d-1c8d35f06b98\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb" Oct 30 16:24:25.748915 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.748887 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsbfz\" (UniqueName: \"kubernetes.io/projected/80c7928f-4dc7-49fc-b98d-1c8d35f06b98-kube-api-access-gsbfz\") pod \"openshift-lws-operator-8465994787-l2bpb\" (UID: \"80c7928f-4dc7-49fc-b98d-1c8d35f06b98\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb" Oct 30 16:24:25.795878 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:25.795846 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb" Oct 30 16:24:26.089830 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:26.089756 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb"] Oct 30 16:24:26.095359 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:24:26.095334 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80c7928f_4dc7_49fc_b98d_1c8d35f06b98.slice/crio-f15a1f7446f41289ec411532957f138d693cf293a76998d6853b8e13b7599ca6 WatchSource:0}: Error finding container f15a1f7446f41289ec411532957f138d693cf293a76998d6853b8e13b7599ca6: Status 404 returned error can't find the container with id f15a1f7446f41289ec411532957f138d693cf293a76998d6853b8e13b7599ca6 Oct 30 16:24:26.209488 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:26.209456 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb" event={"ID":"80c7928f-4dc7-49fc-b98d-1c8d35f06b98","Type":"ContainerStarted","Data":"f15a1f7446f41289ec411532957f138d693cf293a76998d6853b8e13b7599ca6"} Oct 30 16:24:31.229981 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:31.229931 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb" event={"ID":"80c7928f-4dc7-49fc-b98d-1c8d35f06b98","Type":"ContainerStarted","Data":"f65ad081f6a4bc5253686a2bc3d585287bd07ff272ccf88b470ab56e44e51e6c"} Oct 30 16:24:31.271392 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:31.271329 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-l2bpb" podStartSLOduration=1.830805035 podStartE2EDuration="6.271310492s" totalImagesPullingTime="4.440505457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:31.270674311 +0000 UTC m=+233.019263719" watchObservedRunningTime="2026-10-30 16:24:31.271310492 +0000 UTC m=+233.019899877" Oct 30 16:24:31.795595 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:24:31.795556 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:24:36.691035 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.691000 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5"] Oct 30 16:24:36.693864 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.693842 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:36.709809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.709789 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:24:36.709951 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.709936 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:24:36.721502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.721476 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-6gfp9\"" Oct 30 16:24:36.722406 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.722386 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5"] Oct 30 16:24:36.815758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.815724 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5\" (UID: \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:36.815758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.815763 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zjd\" (UniqueName: \"kubernetes.io/projected/fceff575-5450-4b60-b5f8-0bbd7208e5cb-kube-api-access-67zjd\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5\" (UID: \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:36.815956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.815895 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5\" (UID: \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:36.916658 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.916618 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5\" (UID: \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:36.916855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.916673 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zjd\" (UniqueName: \"kubernetes.io/projected/fceff575-5450-4b60-b5f8-0bbd7208e5cb-kube-api-access-67zjd\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5\" (UID: \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:36.916855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.916742 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5\" (UID: \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:36.917120 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.917097 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5\" (UID: \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:36.917202 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.917124 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5\" (UID: \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:36.941235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:36.941176 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zjd\" (UniqueName: \"kubernetes.io/projected/fceff575-5450-4b60-b5f8-0bbd7208e5cb-kube-api-access-67zjd\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5\" (UID: \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:37.006709 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:37.006673 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:37.303291 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:37.303258 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5"] Oct 30 16:24:37.309239 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:24:37.309207 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfceff575_5450_4b60_b5f8_0bbd7208e5cb.slice/crio-5c4ddb6017a1b697bcb054814a0ff7c3282d002ac6a9adb1c34a81a4114653c2 WatchSource:0}: Error finding container 5c4ddb6017a1b697bcb054814a0ff7c3282d002ac6a9adb1c34a81a4114653c2: Status 404 returned error can't find the container with id 5c4ddb6017a1b697bcb054814a0ff7c3282d002ac6a9adb1c34a81a4114653c2 Oct 30 16:24:38.258518 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:38.258483 5838 generic.go:356] "Generic (PLEG): container finished" podID="fceff575-5450-4b60-b5f8-0bbd7208e5cb" containerID="3b1cc681fe548be21112fa9369174c2dfb32cc81b3f06581fa91aa42bd533567" exitCode=0 Oct 30 16:24:38.258870 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:38.258524 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" event={"ID":"fceff575-5450-4b60-b5f8-0bbd7208e5cb","Type":"ContainerDied","Data":"3b1cc681fe548be21112fa9369174c2dfb32cc81b3f06581fa91aa42bd533567"} Oct 30 16:24:38.258870 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:38.258560 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" event={"ID":"fceff575-5450-4b60-b5f8-0bbd7208e5cb","Type":"ContainerStarted","Data":"5c4ddb6017a1b697bcb054814a0ff7c3282d002ac6a9adb1c34a81a4114653c2"} Oct 30 16:24:39.269151 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:39.269107 5838 generic.go:356] "Generic (PLEG): container finished" podID="fceff575-5450-4b60-b5f8-0bbd7208e5cb" containerID="f417c951e2fa7d1bc69b15f0e2475f18775fc52a6be2485d7d490c40264709d4" exitCode=0 Oct 30 16:24:39.269610 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:39.269173 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" event={"ID":"fceff575-5450-4b60-b5f8-0bbd7208e5cb","Type":"ContainerDied","Data":"f417c951e2fa7d1bc69b15f0e2475f18775fc52a6be2485d7d490c40264709d4"} Oct 30 16:24:40.275797 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:40.275760 5838 generic.go:356] "Generic (PLEG): container finished" podID="fceff575-5450-4b60-b5f8-0bbd7208e5cb" containerID="dd6b4b1464f309da87ec5917e06054e5643da3beeac40cfdc8b72642ed2d3d30" exitCode=0 Oct 30 16:24:40.276202 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:40.275840 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" event={"ID":"fceff575-5450-4b60-b5f8-0bbd7208e5cb","Type":"ContainerDied","Data":"dd6b4b1464f309da87ec5917e06054e5643da3beeac40cfdc8b72642ed2d3d30"} Oct 30 16:24:41.402082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:41.402058 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:41.451459 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:41.451422 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-util\" (UniqueName: \"kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-util\") pod \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\" (UID: \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\") " Oct 30 16:24:41.451642 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:41.451515 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-bundle\") pod \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\" (UID: \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\") " Oct 30 16:24:41.452217 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:41.452188 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-bundle" pod "fceff575-5450-4b60-b5f8-0bbd7208e5cb" (UID: "fceff575-5450-4b60-b5f8-0bbd7208e5cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:24:41.456684 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:41.456655 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-util" pod "fceff575-5450-4b60-b5f8-0bbd7208e5cb" (UID: "fceff575-5450-4b60-b5f8-0bbd7208e5cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:24:41.552841 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:41.552747 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fceff575-5450-4b60-b5f8-0bbd7208e5cb-kube-api-access-67zjd\" (UniqueName: \"kubernetes.io/projected/fceff575-5450-4b60-b5f8-0bbd7208e5cb-kube-api-access-67zjd\") pod \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\" (UID: \"fceff575-5450-4b60-b5f8-0bbd7208e5cb\") " Oct 30 16:24:41.553058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:41.552936 5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:24:41.553058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:41.552953 5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fceff575-5450-4b60-b5f8-0bbd7208e5cb-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:24:41.554868 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:41.554841 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fceff575-5450-4b60-b5f8-0bbd7208e5cb-kube-api-access-67zjd" pod "fceff575-5450-4b60-b5f8-0bbd7208e5cb" (UID: "fceff575-5450-4b60-b5f8-0bbd7208e5cb"). InnerVolumeSpecName "kube-api-access-67zjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:24:41.653512 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:41.653472 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67zjd\" (UniqueName: \"kubernetes.io/projected/fceff575-5450-4b60-b5f8-0bbd7208e5cb-kube-api-access-67zjd\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:24:42.285887 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:42.285850 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" event={"ID":"fceff575-5450-4b60-b5f8-0bbd7208e5cb","Type":"ContainerDied","Data":"5c4ddb6017a1b697bcb054814a0ff7c3282d002ac6a9adb1c34a81a4114653c2"} Oct 30 16:24:42.286181 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:42.285891 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be715jgk5" Oct 30 16:24:42.286181 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:42.285900 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4ddb6017a1b697bcb054814a0ff7c3282d002ac6a9adb1c34a81a4114653c2" Oct 30 16:24:51.923440 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.923404 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc"] Oct 30 16:24:51.923847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.923714 5838 cpu_manager.go:414] "containerMap: removing container" podUID="fceff575-5450-4b60-b5f8-0bbd7208e5cb" containerName="extract" Oct 30 16:24:51.923847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.923742 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fceff575-5450-4b60-b5f8-0bbd7208e5cb" containerName="extract" Oct 30 16:24:51.923847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.923768 5838 cpu_manager.go:414] "containerMap: removing container" podUID="fceff575-5450-4b60-b5f8-0bbd7208e5cb" containerName="util" Oct 30 16:24:51.923847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.923772 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fceff575-5450-4b60-b5f8-0bbd7208e5cb" containerName="util" Oct 30 16:24:51.923847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.923800 5838 cpu_manager.go:414] "containerMap: removing container" podUID="fceff575-5450-4b60-b5f8-0bbd7208e5cb" containerName="pull" Oct 30 16:24:51.923847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.923806 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fceff575-5450-4b60-b5f8-0bbd7208e5cb" containerName="pull" Oct 30 16:24:51.924072 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.923864 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="fceff575-5450-4b60-b5f8-0bbd7208e5cb" containerName="extract" Oct 30 16:24:51.926062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.926045 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:51.947225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.947188 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 16:24:51.947378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.947268 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 16:24:51.958586 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.958560 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 16:24:51.958586 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.958576 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 16:24:51.958787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.958560 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 16:24:51.958787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.958615 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7jncl\"" Oct 30 16:24:51.976596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:51.976565 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc"] Oct 30 16:24:52.033584 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.033543 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.033771 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.033593 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-manifests\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.033771 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.033612 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.033871 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.033765 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-metrics-cert\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.033871 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.033822 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxlgk\" (UniqueName: \"kubernetes.io/projected/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-kube-api-access-gxlgk\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.033954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.033918 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-tmp\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.135309 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.135264 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxlgk\" (UniqueName: \"kubernetes.io/projected/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-kube-api-access-gxlgk\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.135309 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.135311 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-tmp\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.135570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.135438 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.135570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.135493 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-manifests\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.135570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.135526 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.135734 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.135593 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-metrics-cert\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.135945 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.135915 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-manifests\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.137590 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.137547 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-tmp\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.137966 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.137941 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-metrics-cert\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.138270 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.138252 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.138336 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.138318 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.159551 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.159516 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxlgk\" (UniqueName: \"kubernetes.io/projected/c64a302c-08f6-4b2b-bd1b-c6bb63a51e27-kube-api-access-gxlgk\") pod \"opendatahub-operator-controller-manager-8dcb75f99-swmnc\" (UID: \"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27\") " pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.241429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.241387 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:52.597116 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:52.597038 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc"] Oct 30 16:24:52.602783 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:24:52.602754 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64a302c_08f6_4b2b_bd1b_c6bb63a51e27.slice/crio-d710e145e847a7eec2fc89d4bef4a2c3ac19b041701c2cee1bb3c3bc7cc18e37 WatchSource:0}: Error finding container d710e145e847a7eec2fc89d4bef4a2c3ac19b041701c2cee1bb3c3bc7cc18e37: Status 404 returned error can't find the container with id d710e145e847a7eec2fc89d4bef4a2c3ac19b041701c2cee1bb3c3bc7cc18e37 Oct 30 16:24:53.334117 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:53.334074 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" event={"ID":"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27","Type":"ContainerStarted","Data":"d710e145e847a7eec2fc89d4bef4a2c3ac19b041701c2cee1bb3c3bc7cc18e37"} Oct 30 16:24:56.348251 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:56.348214 5838 generic.go:356] "Generic (PLEG): container finished" podID="c64a302c-08f6-4b2b-bd1b-c6bb63a51e27" containerID="b49c7cc44258c395ba7b4ef066c7246cd2d7f840aaf75094730fe8a3b6e16953" exitCode=0 Oct 30 16:24:56.348693 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:56.348255 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" event={"ID":"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27","Type":"ContainerDied","Data":"b49c7cc44258c395ba7b4ef066c7246cd2d7f840aaf75094730fe8a3b6e16953"} Oct 30 16:24:57.357000 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:57.356903 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" event={"ID":"c64a302c-08f6-4b2b-bd1b-c6bb63a51e27","Type":"ContainerStarted","Data":"3957bb88dc12e24fab60df691f02335efd29540fed82ad9692f411174b8d6075"} Oct 30 16:24:57.357482 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:57.357291 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:24:57.441960 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:24:57.441905 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" podStartSLOduration=2.744405875 podStartE2EDuration="6.441888455s" totalImagesPullingTime="3.265693342s" totalInitContainerRuntime="431.789238ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:57.430311599 +0000 UTC m=+259.178901009" watchObservedRunningTime="2026-10-30 16:24:57.441888455 +0000 UTC m=+259.190477842" Oct 30 16:25:01.795591 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:25:01.795549 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:25:08.363038 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:08.363007 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8dcb75f99-swmnc" Oct 30 16:25:12.051039 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.051002 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr"] Oct 30 16:25:12.056394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.056370 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" Oct 30 16:25:12.084122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.084091 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 16:25:12.084394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.084106 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 16:25:12.084394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.084159 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 16:25:12.084394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.084221 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5mz6d\"" Oct 30 16:25:12.084394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.084228 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 16:25:12.085863 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.085838 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr"] Oct 30 16:25:12.205888 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.205855 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/035709ff-bb86-42f9-952b-95e17119d171-tls-certs\") pod \"kube-auth-proxy-5dcc5c66cd-7dvjr\" (UID: \"035709ff-bb86-42f9-952b-95e17119d171\") " pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" Oct 30 16:25:12.206082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.205923 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/035709ff-bb86-42f9-952b-95e17119d171-tmp\") pod \"kube-auth-proxy-5dcc5c66cd-7dvjr\" (UID: \"035709ff-bb86-42f9-952b-95e17119d171\") " pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" Oct 30 16:25:12.206082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.206065 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckm57\" (UniqueName: \"kubernetes.io/projected/035709ff-bb86-42f9-952b-95e17119d171-kube-api-access-ckm57\") pod \"kube-auth-proxy-5dcc5c66cd-7dvjr\" (UID: \"035709ff-bb86-42f9-952b-95e17119d171\") " pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" Oct 30 16:25:12.306670 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.306589 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/035709ff-bb86-42f9-952b-95e17119d171-tmp\") pod \"kube-auth-proxy-5dcc5c66cd-7dvjr\" (UID: \"035709ff-bb86-42f9-952b-95e17119d171\") " pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" Oct 30 16:25:12.306670 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.306641 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckm57\" (UniqueName: \"kubernetes.io/projected/035709ff-bb86-42f9-952b-95e17119d171-kube-api-access-ckm57\") pod \"kube-auth-proxy-5dcc5c66cd-7dvjr\" (UID: \"035709ff-bb86-42f9-952b-95e17119d171\") " pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" Oct 30 16:25:12.306884 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.306673 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/035709ff-bb86-42f9-952b-95e17119d171-tls-certs\") pod \"kube-auth-proxy-5dcc5c66cd-7dvjr\" (UID: \"035709ff-bb86-42f9-952b-95e17119d171\") " pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" Oct 30 16:25:12.308954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.308923 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/035709ff-bb86-42f9-952b-95e17119d171-tmp\") pod \"kube-auth-proxy-5dcc5c66cd-7dvjr\" (UID: \"035709ff-bb86-42f9-952b-95e17119d171\") " pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" Oct 30 16:25:12.309179 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.309156 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/035709ff-bb86-42f9-952b-95e17119d171-tls-certs\") pod \"kube-auth-proxy-5dcc5c66cd-7dvjr\" (UID: \"035709ff-bb86-42f9-952b-95e17119d171\") " pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" Oct 30 16:25:12.329163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.329136 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckm57\" (UniqueName: \"kubernetes.io/projected/035709ff-bb86-42f9-952b-95e17119d171-kube-api-access-ckm57\") pod \"kube-auth-proxy-5dcc5c66cd-7dvjr\" (UID: \"035709ff-bb86-42f9-952b-95e17119d171\") " pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" Oct 30 16:25:12.367915 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.367881 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" Oct 30 16:25:12.668115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:12.668034 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr"] Oct 30 16:25:12.673951 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:25:12.673926 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035709ff_bb86_42f9_952b_95e17119d171.slice/crio-e75e8375fae0dad1cf5cc4837988ef8d959c62d9b0936ebb96b695155e0c27f0 WatchSource:0}: Error finding container e75e8375fae0dad1cf5cc4837988ef8d959c62d9b0936ebb96b695155e0c27f0: Status 404 returned error can't find the container with id e75e8375fae0dad1cf5cc4837988ef8d959c62d9b0936ebb96b695155e0c27f0 Oct 30 16:25:13.430052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.430009 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" event={"ID":"035709ff-bb86-42f9-952b-95e17119d171","Type":"ContainerStarted","Data":"e75e8375fae0dad1cf5cc4837988ef8d959c62d9b0936ebb96b695155e0c27f0"} Oct 30 16:25:13.808116 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.807675 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk"] Oct 30 16:25:13.811799 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.811294 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:13.828183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.828150 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 16:25:13.828590 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.828548 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 16:25:13.828715 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.828639 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 16:25:13.839466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.839429 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 16:25:13.839598 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.839517 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qqpz6\"" Oct 30 16:25:13.841755 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.841734 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk"] Oct 30 16:25:13.921077 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.921042 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92465309-5321-4428-b32d-a43c95719ca6-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:13.921301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.921085 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:13.921301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.921130 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g2cx\" (UniqueName: \"kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-kube-api-access-9g2cx\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:13.921301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.921280 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:13.921462 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.921321 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:13.921462 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.921343 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:13.921462 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:13.921425 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.022235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.022197 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.022423 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.022248 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.022423 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.022282 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.022423 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.022314 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.022423 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.022366 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92465309-5321-4428-b32d-a43c95719ca6-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.022630 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.022494 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.022630 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.022545 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2cx\" (UniqueName: \"kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-kube-api-access-9g2cx\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.023194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.023160 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.024873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.024823 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92465309-5321-4428-b32d-a43c95719ca6-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.025189 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.025167 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.025370 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.025343 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.025471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.025452 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.045465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.045438 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.045602 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.045462 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2cx\" (UniqueName: \"kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-kube-api-access-9g2cx\") pod \"istiod-openshift-gateway-7876bfcb74-z8sdk\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.126446 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.126373 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:14.450908 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:14.450831 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk"] Oct 30 16:25:14.456110 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:25:14.456081 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92465309_5321_4428_b32d_a43c95719ca6.slice/crio-991dd2950baa2792b8ff2e59730a2f0245d95bb62e6447b285b1ec994762ae56 WatchSource:0}: Error finding container 991dd2950baa2792b8ff2e59730a2f0245d95bb62e6447b285b1ec994762ae56: Status 404 returned error can't find the container with id 991dd2950baa2792b8ff2e59730a2f0245d95bb62e6447b285b1ec994762ae56 Oct 30 16:25:15.440190 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:15.440102 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" event={"ID":"92465309-5321-4428-b32d-a43c95719ca6","Type":"ContainerStarted","Data":"991dd2950baa2792b8ff2e59730a2f0245d95bb62e6447b285b1ec994762ae56"} Oct 30 16:25:15.441587 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:15.441555 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" event={"ID":"035709ff-bb86-42f9-952b-95e17119d171","Type":"ContainerStarted","Data":"ae3b7f48e4da358dcb28f0407e3631631ecb893c34aca1fdf0367b2f2287e524"} Oct 30 16:25:15.481327 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:15.481257 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5dcc5c66cd-7dvjr" podStartSLOduration=1.070663474 podStartE2EDuration="3.481239354s" totalImagesPullingTime="2.41057588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:15.480513566 +0000 UTC m=+277.229102978" watchObservedRunningTime="2026-10-30 16:25:15.481239354 +0000 UTC m=+277.229828742" Oct 30 16:25:17.238014 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:17.237984 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:25:17.238274 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:17.238046 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:25:17.454457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:17.454416 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" event={"ID":"92465309-5321-4428-b32d-a43c95719ca6","Type":"ContainerStarted","Data":"11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29"} Oct 30 16:25:17.454634 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:17.454550 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:17.521833 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:17.521729 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" podStartSLOduration=1.7430942360000001 podStartE2EDuration="4.521714248s" totalImagesPullingTime="2.778620012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:17.509825936 +0000 UTC m=+279.258415325" watchObservedRunningTime="2026-10-30 16:25:17.521714248 +0000 UTC m=+279.270303632" Oct 30 16:25:18.460250 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.460216 5838 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-z8sdk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:25:18.460688 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.460274 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" podUID="92465309-5321-4428-b32d-a43c95719ca6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:25:18.648886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.648812 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj"] Oct 30 16:25:18.652466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.652441 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.670819 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.670789 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-57lj9\"" Oct 30 16:25:18.680756 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.680725 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj"] Oct 30 16:25:18.760606 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.760559 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2f91bc-56b8-45bb-9b39-876213305120-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.760809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.760621 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2f91bc-56b8-45bb-9b39-876213305120-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.760809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.760723 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.760809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.760756 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.760809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.760780 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.761088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.760883 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.761088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.760909 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.761088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.760930 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2f91bc-56b8-45bb-9b39-876213305120-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.761088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.760956 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5tbm\" (UniqueName: \"kubernetes.io/projected/5c2f91bc-56b8-45bb-9b39-876213305120-kube-api-access-d5tbm\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.861728 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.861691 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2f91bc-56b8-45bb-9b39-876213305120-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.861728 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.861734 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5tbm\" (UniqueName: \"kubernetes.io/projected/5c2f91bc-56b8-45bb-9b39-876213305120-kube-api-access-d5tbm\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.861933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.861781 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2f91bc-56b8-45bb-9b39-876213305120-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.861933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.861803 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2f91bc-56b8-45bb-9b39-876213305120-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.861933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.861824 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.861933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.861853 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.862162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.862019 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.862162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.862098 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.862162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.862125 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.862332 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.862294 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.862412 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.862326 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.862501 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.862480 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.862622 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.862602 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.862784 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.862762 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c2f91bc-56b8-45bb-9b39-876213305120-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.864260 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.864234 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c2f91bc-56b8-45bb-9b39-876213305120-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.864520 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.864500 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c2f91bc-56b8-45bb-9b39-876213305120-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.884596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.884570 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c2f91bc-56b8-45bb-9b39-876213305120-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.884785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.884762 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5tbm\" (UniqueName: \"kubernetes.io/projected/5c2f91bc-56b8-45bb-9b39-876213305120-kube-api-access-d5tbm\") pod \"data-science-gateway-data-science-gateway-class-67df64d57-vdpqj\" (UID: \"5c2f91bc-56b8-45bb-9b39-876213305120\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:18.967896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:18.967861 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:19.288232 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:19.288116 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj"] Oct 30 16:25:19.293182 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:25:19.293151 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2f91bc_56b8_45bb_9b39_876213305120.slice/crio-17af13acc20f0526ff50581e2f8ffa68c374760a466220207b47f717b0798f08 WatchSource:0}: Error finding container 17af13acc20f0526ff50581e2f8ffa68c374760a466220207b47f717b0798f08: Status 404 returned error can't find the container with id 17af13acc20f0526ff50581e2f8ffa68c374760a466220207b47f717b0798f08 Oct 30 16:25:19.467757 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:19.467703 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" event={"ID":"5c2f91bc-56b8-45bb-9b39-876213305120","Type":"ContainerStarted","Data":"17af13acc20f0526ff50581e2f8ffa68c374760a466220207b47f717b0798f08"} Oct 30 16:25:21.459684 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:21.459603 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:25:21.875018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:21.874990 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:25:21.875085 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:21.875061 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:25:21.875119 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:21.875091 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:25:22.482929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:22.482892 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" event={"ID":"5c2f91bc-56b8-45bb-9b39-876213305120","Type":"ContainerStarted","Data":"121a57ae007385e4e677a2e0a28c8992a6edacd9aeca744539fbf0b3e1759c83"} Oct 30 16:25:22.539525 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:22.539469 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" podStartSLOduration=1.960781288 podStartE2EDuration="4.539453317s" totalImagesPullingTime="2.578672029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:22.538506693 +0000 UTC m=+284.287096115" watchObservedRunningTime="2026-10-30 16:25:22.539453317 +0000 UTC m=+284.288042700" Oct 30 16:25:22.968174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:22.968136 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:22.972639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:22.972611 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:23.487679 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:23.487650 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:23.488548 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:23.488526 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67df64d57-vdpqj" Oct 30 16:25:31.796980 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:25:31.796924 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:25:38.413031 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:38.412952 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:25:38.414451 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:25:38.414426 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:26:01.796456 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:26:01.796415 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:26:16.404267 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.404230 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp"] Oct 30 16:26:16.412011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.411981 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.433153 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.433112 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp"] Oct 30 16:26:16.575068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.575023 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb333f2b-2469-4dc5-bb87-2e633640b18b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.575068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.575072 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb333f2b-2469-4dc5-bb87-2e633640b18b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.575353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.575095 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65wjz\" (UniqueName: \"kubernetes.io/projected/bb333f2b-2469-4dc5-bb87-2e633640b18b-kube-api-access-65wjz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.575353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.575261 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bb333f2b-2469-4dc5-bb87-2e633640b18b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.575353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.575319 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bb333f2b-2469-4dc5-bb87-2e633640b18b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.575501 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.575377 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bb333f2b-2469-4dc5-bb87-2e633640b18b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.575501 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.575404 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb333f2b-2469-4dc5-bb87-2e633640b18b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.676077 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.675952 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb333f2b-2469-4dc5-bb87-2e633640b18b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.676077 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.676034 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb333f2b-2469-4dc5-bb87-2e633640b18b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.676077 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.676061 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65wjz\" (UniqueName: \"kubernetes.io/projected/bb333f2b-2469-4dc5-bb87-2e633640b18b-kube-api-access-65wjz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.676387 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.676115 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bb333f2b-2469-4dc5-bb87-2e633640b18b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.676387 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.676147 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bb333f2b-2469-4dc5-bb87-2e633640b18b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.676387 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.676179 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bb333f2b-2469-4dc5-bb87-2e633640b18b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.676387 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.676198 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb333f2b-2469-4dc5-bb87-2e633640b18b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.676985 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.676937 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bb333f2b-2469-4dc5-bb87-2e633640b18b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.678784 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.678753 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb333f2b-2469-4dc5-bb87-2e633640b18b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.678892 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.678852 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb333f2b-2469-4dc5-bb87-2e633640b18b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.679080 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.679063 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bb333f2b-2469-4dc5-bb87-2e633640b18b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.679152 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.679133 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bb333f2b-2469-4dc5-bb87-2e633640b18b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.707880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.707845 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb333f2b-2469-4dc5-bb87-2e633640b18b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.708044 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.708023 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65wjz\" (UniqueName: \"kubernetes.io/projected/bb333f2b-2469-4dc5-bb87-2e633640b18b-kube-api-access-65wjz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vl9xp\" (UID: \"bb333f2b-2469-4dc5-bb87-2e633640b18b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:16.723258 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:16.723224 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:17.043381 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:26:17.043347 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb333f2b_2469_4dc5_bb87_2e633640b18b.slice/crio-1f891c987055166dfbb0e72e9aae7a423703335df1c74ea419cdd398b8eaa0ad WatchSource:0}: Error finding container 1f891c987055166dfbb0e72e9aae7a423703335df1c74ea419cdd398b8eaa0ad: Status 404 returned error can't find the container with id 1f891c987055166dfbb0e72e9aae7a423703335df1c74ea419cdd398b8eaa0ad Oct 30 16:26:17.045786 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:17.045741 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:26:17.045928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:17.045834 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:26:17.045928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:17.045770 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp"] Oct 30 16:26:17.723603 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:17.723549 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" event={"ID":"bb333f2b-2469-4dc5-bb87-2e633640b18b","Type":"ContainerStarted","Data":"b6d78ddbce2c5ceb0c617d351781fd03c47afb8192df09a0daa1d6cbe955f551"} Oct 30 16:26:17.724119 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:17.723632 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" event={"ID":"bb333f2b-2469-4dc5-bb87-2e633640b18b","Type":"ContainerStarted","Data":"1f891c987055166dfbb0e72e9aae7a423703335df1c74ea419cdd398b8eaa0ad"} Oct 30 16:26:17.724119 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:17.723853 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:17.770632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:17.770577 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" podStartSLOduration=1.770563447 podStartE2EDuration="1.770563447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:26:17.769496064 +0000 UTC m=+339.518085451" watchObservedRunningTime="2026-10-30 16:26:17.770563447 +0000 UTC m=+339.519152833" Oct 30 16:26:18.730118 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:18.730086 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vl9xp" Oct 30 16:26:18.816902 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:18.816868 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk"] Oct 30 16:26:18.817276 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:18.817189 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" podUID="92465309-5321-4428-b32d-a43c95719ca6" containerName="discovery" containerID="cri-o://11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29" gracePeriod=30 Oct 30 16:26:19.055573 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.055547 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:26:19.201682 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.201647 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-ca-configmap\") pod \"92465309-5321-4428-b32d-a43c95719ca6\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " Oct 30 16:26:19.201857 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.201704 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-istio-token\" (UniqueName: \"kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-istio-token\") pod \"92465309-5321-4428-b32d-a43c95719ca6\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " Oct 30 16:26:19.201857 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.201728 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-kubeconfig\") pod \"92465309-5321-4428-b32d-a43c95719ca6\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " Oct 30 16:26:19.201992 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.201916 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-dns-cert\") pod \"92465309-5321-4428-b32d-a43c95719ca6\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " Oct 30 16:26:19.202062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.202006 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-kube-api-access-9g2cx\" (UniqueName: \"kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-kube-api-access-9g2cx\") pod \"92465309-5321-4428-b32d-a43c95719ca6\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " Oct 30 16:26:19.202115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.202065 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-cacerts\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-cacerts\") pod \"92465309-5321-4428-b32d-a43c95719ca6\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " Oct 30 16:26:19.202164 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.202101 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-ca-configmap" pod "92465309-5321-4428-b32d-a43c95719ca6" (UID: "92465309-5321-4428-b32d-a43c95719ca6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:26:19.202164 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.202133 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92465309-5321-4428-b32d-a43c95719ca6-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92465309-5321-4428-b32d-a43c95719ca6-local-certs\") pod \"92465309-5321-4428-b32d-a43c95719ca6\" (UID: \"92465309-5321-4428-b32d-a43c95719ca6\") " Oct 30 16:26:19.202415 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.202389 5838 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-ca-configmap\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:26:19.204367 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.204337 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-cacerts" pod "92465309-5321-4428-b32d-a43c95719ca6" (UID: "92465309-5321-4428-b32d-a43c95719ca6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:26:19.204480 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.204424 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-kubeconfig" pod "92465309-5321-4428-b32d-a43c95719ca6" (UID: "92465309-5321-4428-b32d-a43c95719ca6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:26:19.204480 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.204462 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-kube-api-access-9g2cx" pod "92465309-5321-4428-b32d-a43c95719ca6" (UID: "92465309-5321-4428-b32d-a43c95719ca6"). InnerVolumeSpecName "kube-api-access-9g2cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:26:19.204569 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.204538 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-dns-cert" pod "92465309-5321-4428-b32d-a43c95719ca6" (UID: "92465309-5321-4428-b32d-a43c95719ca6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:26:19.204605 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.204578 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92465309-5321-4428-b32d-a43c95719ca6-local-certs" pod "92465309-5321-4428-b32d-a43c95719ca6" (UID: "92465309-5321-4428-b32d-a43c95719ca6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:26:19.204641 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.204607 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-istio-token" pod "92465309-5321-4428-b32d-a43c95719ca6" (UID: "92465309-5321-4428-b32d-a43c95719ca6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:26:19.303466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.303428 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:26:19.303466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.303459 5838 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-kubeconfig\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:26:19.303466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.303468 5838 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-istio-csr-dns-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:26:19.303466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.303478 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g2cx\" (UniqueName: \"kubernetes.io/projected/92465309-5321-4428-b32d-a43c95719ca6-kube-api-access-9g2cx\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:26:19.303736 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.303486 5838 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92465309-5321-4428-b32d-a43c95719ca6-cacerts\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:26:19.303736 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.303495 5838 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92465309-5321-4428-b32d-a43c95719ca6-local-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:26:19.733582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.733486 5838 generic.go:356] "Generic (PLEG): container finished" podID="92465309-5321-4428-b32d-a43c95719ca6" containerID="11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29" exitCode=0 Oct 30 16:26:19.733582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.733534 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" Oct 30 16:26:19.733582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.733568 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" event={"ID":"92465309-5321-4428-b32d-a43c95719ca6","Type":"ContainerDied","Data":"11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29"} Oct 30 16:26:19.734136 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.733622 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk" event={"ID":"92465309-5321-4428-b32d-a43c95719ca6","Type":"ContainerDied","Data":"991dd2950baa2792b8ff2e59730a2f0245d95bb62e6447b285b1ec994762ae56"} Oct 30 16:26:19.734136 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.733651 5838 scope.go:122] "RemoveContainer" containerID="11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29" Oct 30 16:26:19.742719 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.742699 5838 scope.go:122] "RemoveContainer" containerID="11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29" Oct 30 16:26:19.743003 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:26:19.742960 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29\": container with ID starting with 11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29 not found: ID does not exist" containerID="11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29" Oct 30 16:26:19.743068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.743013 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29"} err="failed to get container status \"11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29\": rpc error: code = NotFound desc = could not find container \"11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29\": container with ID starting with 11dcefe309ed3f1fb259d8962f4dabf02362e45a37242c21153f2d727bf30f29 not found: ID does not exist" Oct 30 16:26:19.784065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.784034 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk"] Oct 30 16:26:19.795581 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:19.795550 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-z8sdk"] Oct 30 16:26:20.393985 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:26:20.393923 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92465309-5321-4428-b32d-a43c95719ca6" path="/var/lib/kubelet/pods/92465309-5321-4428-b32d-a43c95719ca6/volumes" Oct 30 16:26:31.795919 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:26:31.795876 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:27:01.795636 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:27:01.795554 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:27:08.596844 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.596809 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn"] Oct 30 16:27:08.597322 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.597154 5838 cpu_manager.go:414] "containerMap: removing container" podUID="92465309-5321-4428-b32d-a43c95719ca6" containerName="discovery" Oct 30 16:27:08.597322 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.597167 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92465309-5321-4428-b32d-a43c95719ca6" containerName="discovery" Oct 30 16:27:08.597322 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.597254 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="92465309-5321-4428-b32d-a43c95719ca6" containerName="discovery" Oct 30 16:27:08.599717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.599696 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:08.624924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.624895 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-f7vt6\"" Oct 30 16:27:08.624924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.624910 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:27:08.625168 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.624910 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:27:08.626457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.626431 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn"] Oct 30 16:27:08.694670 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.694634 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz"] Oct 30 16:27:08.697621 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.697599 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:08.702939 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.702908 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn\" (UID: \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:08.703082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.703018 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn\" (UID: \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:08.703082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.703052 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2xtx\" (UniqueName: \"kubernetes.io/projected/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-kube-api-access-v2xtx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn\" (UID: \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:08.711362 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.711342 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-lpjr2\"" Oct 30 16:27:08.718217 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.718197 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz"] Oct 30 16:27:08.795528 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.795496 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq"] Oct 30 16:27:08.798414 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.798393 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:08.803547 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.803515 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn\" (UID: \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:08.803701 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.803564 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz\" (UID: \"37730d3f-1958-4642-a026-a44f66f7e8b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:08.803701 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.803631 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn\" (UID: \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:08.803701 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.803661 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2xtx\" (UniqueName: \"kubernetes.io/projected/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-kube-api-access-v2xtx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn\" (UID: \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:08.803868 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.803770 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b24fn\" (UniqueName: \"kubernetes.io/projected/37730d3f-1958-4642-a026-a44f66f7e8b5-kube-api-access-b24fn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz\" (UID: \"37730d3f-1958-4642-a026-a44f66f7e8b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:08.803925 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.803866 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz\" (UID: \"37730d3f-1958-4642-a026-a44f66f7e8b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:08.803925 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.803880 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn\" (UID: \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:08.804080 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.804059 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn\" (UID: \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:08.812200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.812158 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-6p7c2\"" Oct 30 16:27:08.818720 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.818700 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq"] Oct 30 16:27:08.826608 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.826583 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2xtx\" (UniqueName: \"kubernetes.io/projected/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-kube-api-access-v2xtx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn\" (UID: \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:08.890505 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.890426 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59"] Oct 30 16:27:08.893236 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.893214 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:08.905043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.905018 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq\" (UID: \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:08.905163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.905056 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b24fn\" (UniqueName: \"kubernetes.io/projected/37730d3f-1958-4642-a026-a44f66f7e8b5-kube-api-access-b24fn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz\" (UID: \"37730d3f-1958-4642-a026-a44f66f7e8b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:08.905163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.905080 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq\" (UID: \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:08.905235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.905177 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz\" (UID: \"37730d3f-1958-4642-a026-a44f66f7e8b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:08.905235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.905200 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz\" (UID: \"37730d3f-1958-4642-a026-a44f66f7e8b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:08.905235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.905225 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qgh\" (UniqueName: \"kubernetes.io/projected/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-kube-api-access-d5qgh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq\" (UID: \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:08.905535 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.905514 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz\" (UID: \"37730d3f-1958-4642-a026-a44f66f7e8b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:08.905575 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.905532 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz\" (UID: \"37730d3f-1958-4642-a026-a44f66f7e8b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:08.907592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.907576 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-zdg8l\"" Oct 30 16:27:08.912729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.912714 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:08.914210 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.914191 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59"] Oct 30 16:27:08.928617 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:08.928589 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b24fn\" (UniqueName: \"kubernetes.io/projected/37730d3f-1958-4642-a026-a44f66f7e8b5-kube-api-access-b24fn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz\" (UID: \"37730d3f-1958-4642-a026-a44f66f7e8b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:09.006579 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.006546 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qgh\" (UniqueName: \"kubernetes.io/projected/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-kube-api-access-d5qgh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq\" (UID: \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:09.006579 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.006586 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59\" (UID: \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:09.006777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.006624 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts7q5\" (UniqueName: \"kubernetes.io/projected/c045cc79-203f-46ec-96d2-4b20afb0bdc8-kube-api-access-ts7q5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59\" (UID: \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:09.006777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.006644 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq\" (UID: \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:09.006777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.006662 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59\" (UID: \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:09.006777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.006681 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq\" (UID: \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:09.007067 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.007043 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq\" (UID: \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:09.007122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.007104 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq\" (UID: \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:09.011483 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.011458 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:09.029994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.029937 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qgh\" (UniqueName: \"kubernetes.io/projected/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-kube-api-access-d5qgh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq\" (UID: \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:09.107558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.107512 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts7q5\" (UniqueName: \"kubernetes.io/projected/c045cc79-203f-46ec-96d2-4b20afb0bdc8-kube-api-access-ts7q5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59\" (UID: \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:09.107735 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.107563 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59\" (UID: \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:09.107735 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.107641 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59\" (UID: \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:09.108011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.107987 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59\" (UID: \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:09.108106 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.108089 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59\" (UID: \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:09.111399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.111377 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:09.130763 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.130741 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts7q5\" (UniqueName: \"kubernetes.io/projected/c045cc79-203f-46ec-96d2-4b20afb0bdc8-kube-api-access-ts7q5\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59\" (UID: \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:09.207334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.207297 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:09.235869 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.235835 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn"] Oct 30 16:27:09.239271 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:27:09.239245 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5a1339_6f7d_4fe6_a6da_8d8c498435c9.slice/crio-31159e482778981167668effa3a720e9b45d2203e2c1b4109fb8fd3823bdb631 WatchSource:0}: Error finding container 31159e482778981167668effa3a720e9b45d2203e2c1b4109fb8fd3823bdb631: Status 404 returned error can't find the container with id 31159e482778981167668effa3a720e9b45d2203e2c1b4109fb8fd3823bdb631 Oct 30 16:27:09.311101 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.311074 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz"] Oct 30 16:27:09.315863 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:27:09.315838 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37730d3f_1958_4642_a026_a44f66f7e8b5.slice/crio-284ba1e8dbefb751432e224974038fe85ec244dbfd5a15d76481f99951e1da65 WatchSource:0}: Error finding container 284ba1e8dbefb751432e224974038fe85ec244dbfd5a15d76481f99951e1da65: Status 404 returned error can't find the container with id 284ba1e8dbefb751432e224974038fe85ec244dbfd5a15d76481f99951e1da65 Oct 30 16:27:09.405313 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.405204 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq"] Oct 30 16:27:09.414182 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:27:09.414154 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec1f8fa_d5c7_4185_ab98_a1b9579469b0.slice/crio-66fbe054f0a73b3e5191268c43b9475ecfbe4a54b2f531d53f553f957c347810 WatchSource:0}: Error finding container 66fbe054f0a73b3e5191268c43b9475ecfbe4a54b2f531d53f553f957c347810: Status 404 returned error can't find the container with id 66fbe054f0a73b3e5191268c43b9475ecfbe4a54b2f531d53f553f957c347810 Oct 30 16:27:09.505506 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.505477 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59"] Oct 30 16:27:09.551918 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:27:09.551891 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc045cc79_203f_46ec_96d2_4b20afb0bdc8.slice/crio-af8f62c5de5e7747eab60aa0d4e3d3eb67c8c37217a4fec36bb2cd7562d7f419 WatchSource:0}: Error finding container af8f62c5de5e7747eab60aa0d4e3d3eb67c8c37217a4fec36bb2cd7562d7f419: Status 404 returned error can't find the container with id af8f62c5de5e7747eab60aa0d4e3d3eb67c8c37217a4fec36bb2cd7562d7f419 Oct 30 16:27:09.953211 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.953121 5838 generic.go:356] "Generic (PLEG): container finished" podID="c045cc79-203f-46ec-96d2-4b20afb0bdc8" containerID="1d798d1ae0fb4b6f38cac08c985a4241a8f923cd41fe4d25a914ed47ff1ad35f" exitCode=0 Oct 30 16:27:09.953211 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.953197 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" event={"ID":"c045cc79-203f-46ec-96d2-4b20afb0bdc8","Type":"ContainerDied","Data":"1d798d1ae0fb4b6f38cac08c985a4241a8f923cd41fe4d25a914ed47ff1ad35f"} Oct 30 16:27:09.953660 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.953286 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" event={"ID":"c045cc79-203f-46ec-96d2-4b20afb0bdc8","Type":"ContainerStarted","Data":"af8f62c5de5e7747eab60aa0d4e3d3eb67c8c37217a4fec36bb2cd7562d7f419"} Oct 30 16:27:09.954550 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.954533 5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:27:09.954838 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.954812 5838 generic.go:356] "Generic (PLEG): container finished" podID="1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" containerID="d95a24eca0b145f37c296401172a3ea32a4e6a687a1e85dd9c9589635293f0c5" exitCode=0 Oct 30 16:27:09.954944 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.954899 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" event={"ID":"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0","Type":"ContainerDied","Data":"d95a24eca0b145f37c296401172a3ea32a4e6a687a1e85dd9c9589635293f0c5"} Oct 30 16:27:09.955036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.954944 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" event={"ID":"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0","Type":"ContainerStarted","Data":"66fbe054f0a73b3e5191268c43b9475ecfbe4a54b2f531d53f553f957c347810"} Oct 30 16:27:09.956691 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.956666 5838 generic.go:356] "Generic (PLEG): container finished" podID="4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" containerID="5c6a56391a5ee53e56d994e399e17d1019068179ee9cc6227b4ee4132737e357" exitCode=0 Oct 30 16:27:09.956791 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.956760 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" event={"ID":"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9","Type":"ContainerDied","Data":"5c6a56391a5ee53e56d994e399e17d1019068179ee9cc6227b4ee4132737e357"} Oct 30 16:27:09.956927 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.956815 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" event={"ID":"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9","Type":"ContainerStarted","Data":"31159e482778981167668effa3a720e9b45d2203e2c1b4109fb8fd3823bdb631"} Oct 30 16:27:09.958588 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.958549 5838 generic.go:356] "Generic (PLEG): container finished" podID="37730d3f-1958-4642-a026-a44f66f7e8b5" containerID="5135abd476bd7238332270969132defc8c497052568afba91bd2da264f4a0572" exitCode=0 Oct 30 16:27:09.958588 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.958579 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" event={"ID":"37730d3f-1958-4642-a026-a44f66f7e8b5","Type":"ContainerDied","Data":"5135abd476bd7238332270969132defc8c497052568afba91bd2da264f4a0572"} Oct 30 16:27:09.958688 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:09.958617 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" event={"ID":"37730d3f-1958-4642-a026-a44f66f7e8b5","Type":"ContainerStarted","Data":"284ba1e8dbefb751432e224974038fe85ec244dbfd5a15d76481f99951e1da65"} Oct 30 16:27:11.969844 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:11.969810 5838 generic.go:356] "Generic (PLEG): container finished" podID="1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" containerID="c31595696abe4310345e672d0db7f73d581ed5a25421edb00f451b15fd9f4c10" exitCode=0 Oct 30 16:27:11.970275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:11.969890 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" event={"ID":"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0","Type":"ContainerDied","Data":"c31595696abe4310345e672d0db7f73d581ed5a25421edb00f451b15fd9f4c10"} Oct 30 16:27:11.971910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:11.971887 5838 generic.go:356] "Generic (PLEG): container finished" podID="4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" containerID="fd302dfaf28bd56e4a5f679d32f222d83a6d6d3eda8dd7e844e212e5261f09d9" exitCode=0 Oct 30 16:27:11.972036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:11.971947 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" event={"ID":"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9","Type":"ContainerDied","Data":"fd302dfaf28bd56e4a5f679d32f222d83a6d6d3eda8dd7e844e212e5261f09d9"} Oct 30 16:27:12.978708 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:12.978649 5838 generic.go:356] "Generic (PLEG): container finished" podID="4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" containerID="a12b1549521bab0ec90379f020d397d6ffff25edb4b2b3a8805849d3090c3258" exitCode=0 Oct 30 16:27:12.979159 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:12.978734 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" event={"ID":"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9","Type":"ContainerDied","Data":"a12b1549521bab0ec90379f020d397d6ffff25edb4b2b3a8805849d3090c3258"} Oct 30 16:27:12.980765 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:12.980737 5838 generic.go:356] "Generic (PLEG): container finished" podID="1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" containerID="ed3c606a688e717c917ad16c475f6936265a0fc55b1eb69c14e908eb9d1aec0f" exitCode=0 Oct 30 16:27:12.980877 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:12.980785 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" event={"ID":"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0","Type":"ContainerDied","Data":"ed3c606a688e717c917ad16c475f6936265a0fc55b1eb69c14e908eb9d1aec0f"} Oct 30 16:27:14.109209 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.109182 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:14.140535 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.140507 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:14.255349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.255264 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-bundle\") pod \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\" (UID: \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\") " Oct 30 16:27:14.255349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.255308 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-kube-api-access-d5qgh\" (UniqueName: \"kubernetes.io/projected/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-kube-api-access-d5qgh\") pod \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\" (UID: \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\") " Oct 30 16:27:14.255587 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.255362 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-bundle\") pod \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\" (UID: \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\") " Oct 30 16:27:14.255587 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.255395 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-kube-api-access-v2xtx\" (UniqueName: \"kubernetes.io/projected/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-kube-api-access-v2xtx\") pod \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\" (UID: \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\") " Oct 30 16:27:14.255587 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.255424 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-util\" (UniqueName: \"kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-util\") pod \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\" (UID: \"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9\") " Oct 30 16:27:14.255587 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.255500 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-util\" (UniqueName: \"kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-util\") pod \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\" (UID: \"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0\") " Oct 30 16:27:14.255925 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.255895 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-bundle" pod "1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" (UID: "1ec1f8fa-d5c7-4185-ab98-a1b9579469b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:14.256253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.256205 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-bundle" pod "4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" (UID: "4a5a1339-6f7d-4fe6-a6da-8d8c498435c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:14.257612 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.257578 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-kube-api-access-v2xtx" pod "4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" (UID: "4a5a1339-6f7d-4fe6-a6da-8d8c498435c9"). InnerVolumeSpecName "kube-api-access-v2xtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:27:14.257704 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.257637 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-kube-api-access-d5qgh" pod "1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" (UID: "1ec1f8fa-d5c7-4185-ab98-a1b9579469b0"). InnerVolumeSpecName "kube-api-access-d5qgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:27:14.264417 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.264383 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-util" pod "1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" (UID: "1ec1f8fa-d5c7-4185-ab98-a1b9579469b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:14.264564 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.264544 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-util" pod "4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" (UID: "4a5a1339-6f7d-4fe6-a6da-8d8c498435c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:14.356980 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.356939 5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:27:14.357127 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.357000 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5qgh\" (UniqueName: \"kubernetes.io/projected/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-kube-api-access-d5qgh\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:27:14.357127 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.357017 5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:27:14.357127 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.357029 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2xtx\" (UniqueName: \"kubernetes.io/projected/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-kube-api-access-v2xtx\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:27:14.357127 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.357041 5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a5a1339-6f7d-4fe6-a6da-8d8c498435c9-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:27:14.357127 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.357053 5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ec1f8fa-d5c7-4185-ab98-a1b9579469b0-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:27:14.991814 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.991787 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" Oct 30 16:27:14.992018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.991808 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7r6zrq" event={"ID":"1ec1f8fa-d5c7-4185-ab98-a1b9579469b0","Type":"ContainerDied","Data":"66fbe054f0a73b3e5191268c43b9475ecfbe4a54b2f531d53f553f957c347810"} Oct 30 16:27:14.992018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.991870 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66fbe054f0a73b3e5191268c43b9475ecfbe4a54b2f531d53f553f957c347810" Oct 30 16:27:14.993808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.993777 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" event={"ID":"4a5a1339-6f7d-4fe6-a6da-8d8c498435c9","Type":"ContainerDied","Data":"31159e482778981167668effa3a720e9b45d2203e2c1b4109fb8fd3823bdb631"} Oct 30 16:27:14.993996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.993832 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31159e482778981167668effa3a720e9b45d2203e2c1b4109fb8fd3823bdb631" Oct 30 16:27:14.993996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:14.993795 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aspmmn" Oct 30 16:27:19.633628 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:27:19.632487 5838 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37730d3f_1958_4642_a026_a44f66f7e8b5.slice/crio-conmon-882552ff113bbbaa6f011fce44ff1538da1baf430084a19fb8a488858ed10eea.scope\": RecentStats: unable to find data in memory cache]" Oct 30 16:27:20.017356 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:20.017325 5838 generic.go:356] "Generic (PLEG): container finished" podID="37730d3f-1958-4642-a026-a44f66f7e8b5" containerID="882552ff113bbbaa6f011fce44ff1538da1baf430084a19fb8a488858ed10eea" exitCode=0 Oct 30 16:27:20.017521 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:20.017417 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" event={"ID":"37730d3f-1958-4642-a026-a44f66f7e8b5","Type":"ContainerDied","Data":"882552ff113bbbaa6f011fce44ff1538da1baf430084a19fb8a488858ed10eea"} Oct 30 16:27:20.019268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:20.019242 5838 generic.go:356] "Generic (PLEG): container finished" podID="c045cc79-203f-46ec-96d2-4b20afb0bdc8" containerID="46cb560b82ee933f10f12b9165e7984a6da632cfa97d907d3f3c8f1a9de63f96" exitCode=0 Oct 30 16:27:20.019361 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:20.019307 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" event={"ID":"c045cc79-203f-46ec-96d2-4b20afb0bdc8","Type":"ContainerDied","Data":"46cb560b82ee933f10f12b9165e7984a6da632cfa97d907d3f3c8f1a9de63f96"} Oct 30 16:27:21.025930 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:21.025898 5838 generic.go:356] "Generic (PLEG): container finished" podID="37730d3f-1958-4642-a026-a44f66f7e8b5" containerID="0e1e4c93aa8e8e99e9801ec70da8b14fb7a570f1c67f2401bdaedb77e730d437" exitCode=0 Oct 30 16:27:21.026331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:21.025943 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" event={"ID":"37730d3f-1958-4642-a026-a44f66f7e8b5","Type":"ContainerDied","Data":"0e1e4c93aa8e8e99e9801ec70da8b14fb7a570f1c67f2401bdaedb77e730d437"} Oct 30 16:27:21.028152 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:21.028131 5838 generic.go:356] "Generic (PLEG): container finished" podID="c045cc79-203f-46ec-96d2-4b20afb0bdc8" containerID="e1bf12c31af785097fdb55f2e3922f41679ffb4d9946c72588487b5836c0bd11" exitCode=0 Oct 30 16:27:21.028223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:21.028210 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" event={"ID":"c045cc79-203f-46ec-96d2-4b20afb0bdc8","Type":"ContainerDied","Data":"e1bf12c31af785097fdb55f2e3922f41679ffb4d9946c72588487b5836c0bd11"} Oct 30 16:27:22.156578 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.156551 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:22.189263 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.189236 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:22.224926 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.224896 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37730d3f-1958-4642-a026-a44f66f7e8b5-kube-api-access-b24fn\" (UniqueName: \"kubernetes.io/projected/37730d3f-1958-4642-a026-a44f66f7e8b5-kube-api-access-b24fn\") pod \"37730d3f-1958-4642-a026-a44f66f7e8b5\" (UID: \"37730d3f-1958-4642-a026-a44f66f7e8b5\") " Oct 30 16:27:22.225112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.224941 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-bundle\") pod \"37730d3f-1958-4642-a026-a44f66f7e8b5\" (UID: \"37730d3f-1958-4642-a026-a44f66f7e8b5\") " Oct 30 16:27:22.225112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.224964 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c045cc79-203f-46ec-96d2-4b20afb0bdc8-kube-api-access-ts7q5\" (UniqueName: \"kubernetes.io/projected/c045cc79-203f-46ec-96d2-4b20afb0bdc8-kube-api-access-ts7q5\") pod \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\" (UID: \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\") " Oct 30 16:27:22.225112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.225007 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-bundle\") pod \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\" (UID: \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\") " Oct 30 16:27:22.225112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.225056 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-util\" (UniqueName: \"kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-util\") pod \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\" (UID: \"c045cc79-203f-46ec-96d2-4b20afb0bdc8\") " Oct 30 16:27:22.225112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.225079 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-util\" (UniqueName: \"kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-util\") pod \"37730d3f-1958-4642-a026-a44f66f7e8b5\" (UID: \"37730d3f-1958-4642-a026-a44f66f7e8b5\") " Oct 30 16:27:22.225621 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.225589 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-bundle" pod "c045cc79-203f-46ec-96d2-4b20afb0bdc8" (UID: "c045cc79-203f-46ec-96d2-4b20afb0bdc8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:22.225737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.225595 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-bundle" pod "37730d3f-1958-4642-a026-a44f66f7e8b5" (UID: "37730d3f-1958-4642-a026-a44f66f7e8b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:22.227232 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.227210 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c045cc79-203f-46ec-96d2-4b20afb0bdc8-kube-api-access-ts7q5" pod "c045cc79-203f-46ec-96d2-4b20afb0bdc8" (UID: "c045cc79-203f-46ec-96d2-4b20afb0bdc8"). InnerVolumeSpecName "kube-api-access-ts7q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:27:22.227427 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.227402 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37730d3f-1958-4642-a026-a44f66f7e8b5-kube-api-access-b24fn" pod "37730d3f-1958-4642-a026-a44f66f7e8b5" (UID: "37730d3f-1958-4642-a026-a44f66f7e8b5"). InnerVolumeSpecName "kube-api-access-b24fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:27:22.230463 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.230430 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-util" pod "c045cc79-203f-46ec-96d2-4b20afb0bdc8" (UID: "c045cc79-203f-46ec-96d2-4b20afb0bdc8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:22.232370 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.232343 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-util" pod "37730d3f-1958-4642-a026-a44f66f7e8b5" (UID: "37730d3f-1958-4642-a026-a44f66f7e8b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:22.326005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.325894 5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:27:22.326005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.325924 5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c045cc79-203f-46ec-96d2-4b20afb0bdc8-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:27:22.326005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.325933 5838 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:27:22.326005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.325941 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b24fn\" (UniqueName: \"kubernetes.io/projected/37730d3f-1958-4642-a026-a44f66f7e8b5-kube-api-access-b24fn\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:27:22.326005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.325951 5838 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37730d3f-1958-4642-a026-a44f66f7e8b5-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:27:22.326005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:22.325961 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts7q5\" (UniqueName: \"kubernetes.io/projected/c045cc79-203f-46ec-96d2-4b20afb0bdc8-kube-api-access-ts7q5\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:27:23.039332 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:23.039302 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" Oct 30 16:27:23.039503 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:23.039337 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ktl59" event={"ID":"c045cc79-203f-46ec-96d2-4b20afb0bdc8","Type":"ContainerDied","Data":"af8f62c5de5e7747eab60aa0d4e3d3eb67c8c37217a4fec36bb2cd7562d7f419"} Oct 30 16:27:23.039503 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:23.039394 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af8f62c5de5e7747eab60aa0d4e3d3eb67c8c37217a4fec36bb2cd7562d7f419" Oct 30 16:27:23.041369 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:23.041345 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" event={"ID":"37730d3f-1958-4642-a026-a44f66f7e8b5","Type":"ContainerDied","Data":"284ba1e8dbefb751432e224974038fe85ec244dbfd5a15d76481f99951e1da65"} Oct 30 16:27:23.041475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:23.041387 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="284ba1e8dbefb751432e224974038fe85ec244dbfd5a15d76481f99951e1da65" Oct 30 16:27:23.041475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:23.041395 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj27zz" Oct 30 16:27:31.795988 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:27:31.795939 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:27:37.953795 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.953757 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg"] Oct 30 16:27:37.954338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954234 5838 cpu_manager.go:414] "containerMap: removing container" podUID="1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" containerName="util" Oct 30 16:27:37.954338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954257 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" containerName="util" Oct 30 16:27:37.954338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954283 5838 cpu_manager.go:414] "containerMap: removing container" podUID="c045cc79-203f-46ec-96d2-4b20afb0bdc8" containerName="pull" Oct 30 16:27:37.954338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954289 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c045cc79-203f-46ec-96d2-4b20afb0bdc8" containerName="pull" Oct 30 16:27:37.954338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954300 5838 cpu_manager.go:414] "containerMap: removing container" podUID="37730d3f-1958-4642-a026-a44f66f7e8b5" containerName="util" Oct 30 16:27:37.954338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954305 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37730d3f-1958-4642-a026-a44f66f7e8b5" containerName="util" Oct 30 16:27:37.954338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954313 5838 cpu_manager.go:414] "containerMap: removing container" podUID="1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" containerName="pull" Oct 30 16:27:37.954338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954320 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" containerName="pull" Oct 30 16:27:37.954338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954335 5838 cpu_manager.go:414] "containerMap: removing container" podUID="37730d3f-1958-4642-a026-a44f66f7e8b5" containerName="extract" Oct 30 16:27:37.954338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954341 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37730d3f-1958-4642-a026-a44f66f7e8b5" containerName="extract" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954354 5838 cpu_manager.go:414] "containerMap: removing container" podUID="37730d3f-1958-4642-a026-a44f66f7e8b5" containerName="pull" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954359 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37730d3f-1958-4642-a026-a44f66f7e8b5" containerName="pull" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954368 5838 cpu_manager.go:414] "containerMap: removing container" podUID="4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" containerName="util" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954372 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" containerName="util" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954378 5838 cpu_manager.go:414] "containerMap: removing container" podUID="1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" containerName="extract" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954383 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" containerName="extract" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954389 5838 cpu_manager.go:414] "containerMap: removing container" podUID="4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" containerName="extract" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954393 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" containerName="extract" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954400 5838 cpu_manager.go:414] "containerMap: removing container" podUID="c045cc79-203f-46ec-96d2-4b20afb0bdc8" containerName="util" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954404 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c045cc79-203f-46ec-96d2-4b20afb0bdc8" containerName="util" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954409 5838 cpu_manager.go:414] "containerMap: removing container" podUID="4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" containerName="pull" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954413 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" containerName="pull" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954418 5838 cpu_manager.go:414] "containerMap: removing container" podUID="c045cc79-203f-46ec-96d2-4b20afb0bdc8" containerName="extract" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954423 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c045cc79-203f-46ec-96d2-4b20afb0bdc8" containerName="extract" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954488 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="37730d3f-1958-4642-a026-a44f66f7e8b5" containerName="extract" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954499 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec1f8fa-d5c7-4185-ab98-a1b9579469b0" containerName="extract" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954504 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5a1339-6f7d-4fe6-a6da-8d8c498435c9" containerName="extract" Oct 30 16:27:37.954717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.954510 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="c045cc79-203f-46ec-96d2-4b20afb0bdc8" containerName="extract" Oct 30 16:27:37.958652 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.958629 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg" Oct 30 16:27:37.986469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.986424 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 16:27:37.986469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.986430 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-j72tb\"" Oct 30 16:27:37.986469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.986460 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 16:27:37.986768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.986431 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 16:27:37.988241 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:37.988213 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg"] Oct 30 16:27:38.059816 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:38.059774 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djjz\" (UniqueName: \"kubernetes.io/projected/3bcaa3b3-de70-4a7c-82f6-5ec295abcfee-kube-api-access-8djjz\") pod \"dns-operator-controller-manager-65bfd79598-x5dpg\" (UID: \"3bcaa3b3-de70-4a7c-82f6-5ec295abcfee\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg" Oct 30 16:27:38.160325 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:38.160289 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djjz\" (UniqueName: \"kubernetes.io/projected/3bcaa3b3-de70-4a7c-82f6-5ec295abcfee-kube-api-access-8djjz\") pod \"dns-operator-controller-manager-65bfd79598-x5dpg\" (UID: \"3bcaa3b3-de70-4a7c-82f6-5ec295abcfee\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg" Oct 30 16:27:38.183113 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:38.183085 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djjz\" (UniqueName: \"kubernetes.io/projected/3bcaa3b3-de70-4a7c-82f6-5ec295abcfee-kube-api-access-8djjz\") pod \"dns-operator-controller-manager-65bfd79598-x5dpg\" (UID: \"3bcaa3b3-de70-4a7c-82f6-5ec295abcfee\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg" Oct 30 16:27:38.272206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:38.272168 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg" Oct 30 16:27:38.568965 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:38.568878 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg"] Oct 30 16:27:38.573856 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:27:38.573823 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcaa3b3_de70_4a7c_82f6_5ec295abcfee.slice/crio-21a386368d71cd79a71fcfd0edcf83879650951e40b6d71aa53b2be4889843dc WatchSource:0}: Error finding container 21a386368d71cd79a71fcfd0edcf83879650951e40b6d71aa53b2be4889843dc: Status 404 returned error can't find the container with id 21a386368d71cd79a71fcfd0edcf83879650951e40b6d71aa53b2be4889843dc Oct 30 16:27:39.113601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:39.113562 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg" event={"ID":"3bcaa3b3-de70-4a7c-82f6-5ec295abcfee","Type":"ContainerStarted","Data":"21a386368d71cd79a71fcfd0edcf83879650951e40b6d71aa53b2be4889843dc"} Oct 30 16:27:41.210692 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:41.210644 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6"] Oct 30 16:27:41.214727 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:41.214701 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:27:41.240346 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:41.240308 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-24ss9\"" Oct 30 16:27:41.240346 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:41.240340 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 16:27:41.242028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:41.241948 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6"] Oct 30 16:27:41.288076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:41.288052 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbkgd\" (UniqueName: \"kubernetes.io/projected/1b0740dd-2f85-43d5-afa1-5d764ce4ba75-kube-api-access-fbkgd\") pod \"limitador-operator-controller-manager-856dd584c5-fhdv6\" (UID: \"1b0740dd-2f85-43d5-afa1-5d764ce4ba75\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:27:41.389027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:41.388960 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbkgd\" (UniqueName: \"kubernetes.io/projected/1b0740dd-2f85-43d5-afa1-5d764ce4ba75-kube-api-access-fbkgd\") pod \"limitador-operator-controller-manager-856dd584c5-fhdv6\" (UID: \"1b0740dd-2f85-43d5-afa1-5d764ce4ba75\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:27:41.412394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:41.412361 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbkgd\" (UniqueName: \"kubernetes.io/projected/1b0740dd-2f85-43d5-afa1-5d764ce4ba75-kube-api-access-fbkgd\") pod \"limitador-operator-controller-manager-856dd584c5-fhdv6\" (UID: \"1b0740dd-2f85-43d5-afa1-5d764ce4ba75\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:27:41.530925 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:41.530882 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:27:41.854150 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:41.854043 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6"] Oct 30 16:27:41.858780 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:27:41.858752 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0740dd_2f85_43d5_afa1_5d764ce4ba75.slice/crio-1c8b95a644860844b813f9c8cea2c3dd756c4118cd1300f281f8ffcb9458d441 WatchSource:0}: Error finding container 1c8b95a644860844b813f9c8cea2c3dd756c4118cd1300f281f8ffcb9458d441: Status 404 returned error can't find the container with id 1c8b95a644860844b813f9c8cea2c3dd756c4118cd1300f281f8ffcb9458d441 Oct 30 16:27:42.131796 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:42.131675 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg" event={"ID":"3bcaa3b3-de70-4a7c-82f6-5ec295abcfee","Type":"ContainerStarted","Data":"565d78e82591c33b42921299aef89c72f3eb9beafa8c99d254c1e8e09d438b6a"} Oct 30 16:27:42.131796 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:42.131759 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg" Oct 30 16:27:42.133090 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:42.133065 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" event={"ID":"1b0740dd-2f85-43d5-afa1-5d764ce4ba75","Type":"ContainerStarted","Data":"1c8b95a644860844b813f9c8cea2c3dd756c4118cd1300f281f8ffcb9458d441"} Oct 30 16:27:42.177989 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:42.177930 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg" podStartSLOduration=2.534348118 podStartE2EDuration="5.177916635s" totalImagesPullingTime="2.643568517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:42.176639932 +0000 UTC m=+423.925229320" watchObservedRunningTime="2026-10-30 16:27:42.177916635 +0000 UTC m=+423.926506021" Oct 30 16:27:43.144907 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:43.144858 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" event={"ID":"1b0740dd-2f85-43d5-afa1-5d764ce4ba75","Type":"ContainerStarted","Data":"7364b444fedb5651e6ba5cf933e411ffcb46ed941912b2d25b018d378a6a6282"} Oct 30 16:27:43.145342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:43.145153 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:27:43.188708 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:43.188655 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" podStartSLOduration=0.987337387 podStartE2EDuration="2.188641849s" totalImagesPullingTime="1.201304462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:43.187307151 +0000 UTC m=+424.935896538" watchObservedRunningTime="2026-10-30 16:27:43.188641849 +0000 UTC m=+424.937231237" Oct 30 16:27:45.098107 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:45.098067 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-srmxg"] Oct 30 16:27:45.103243 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:45.103220 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-srmxg" Oct 30 16:27:45.131923 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:45.131890 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nl72l\"" Oct 30 16:27:45.145455 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:45.145424 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-srmxg"] Oct 30 16:27:45.229726 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:45.229693 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8qfr\" (UniqueName: \"kubernetes.io/projected/e32488e8-fdae-4b20-906b-2bb8d48be984-kube-api-access-p8qfr\") pod \"authorino-operator-68d6dd76dc-srmxg\" (UID: \"e32488e8-fdae-4b20-906b-2bb8d48be984\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-srmxg" Oct 30 16:27:45.330342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:45.330299 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8qfr\" (UniqueName: \"kubernetes.io/projected/e32488e8-fdae-4b20-906b-2bb8d48be984-kube-api-access-p8qfr\") pod \"authorino-operator-68d6dd76dc-srmxg\" (UID: \"e32488e8-fdae-4b20-906b-2bb8d48be984\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-srmxg" Oct 30 16:27:45.353188 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:45.353114 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8qfr\" (UniqueName: \"kubernetes.io/projected/e32488e8-fdae-4b20-906b-2bb8d48be984-kube-api-access-p8qfr\") pod \"authorino-operator-68d6dd76dc-srmxg\" (UID: \"e32488e8-fdae-4b20-906b-2bb8d48be984\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-srmxg" Oct 30 16:27:45.415958 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:45.415920 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-srmxg" Oct 30 16:27:45.713708 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:45.713592 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-srmxg"] Oct 30 16:27:45.718722 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:27:45.718693 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32488e8_fdae_4b20_906b_2bb8d48be984.slice/crio-cfaf02710f6f39842c47acd92a6019d983ec9aef009beda5d71d654b40dc6056 WatchSource:0}: Error finding container cfaf02710f6f39842c47acd92a6019d983ec9aef009beda5d71d654b40dc6056: Status 404 returned error can't find the container with id cfaf02710f6f39842c47acd92a6019d983ec9aef009beda5d71d654b40dc6056 Oct 30 16:27:46.162040 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:46.162002 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-srmxg" event={"ID":"e32488e8-fdae-4b20-906b-2bb8d48be984","Type":"ContainerStarted","Data":"cfaf02710f6f39842c47acd92a6019d983ec9aef009beda5d71d654b40dc6056"} Oct 30 16:27:48.174234 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:48.174199 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-srmxg" event={"ID":"e32488e8-fdae-4b20-906b-2bb8d48be984","Type":"ContainerStarted","Data":"364f3db538ec9d2cb34407c59a9887c83a3f25018ce05e4cdf7920958be0a246"} Oct 30 16:27:48.174876 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:48.174277 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-srmxg" Oct 30 16:27:48.220302 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:48.220251 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-srmxg" podStartSLOduration=1.658347285 podStartE2EDuration="3.220236778s" totalImagesPullingTime="1.561889493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:48.220125383 +0000 UTC m=+429.968714790" watchObservedRunningTime="2026-10-30 16:27:48.220236778 +0000 UTC m=+429.968826165" Oct 30 16:27:53.148080 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:53.148047 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-x5dpg" Oct 30 16:27:54.152868 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:54.152831 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:27:59.182419 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:27:59.182384 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-srmxg" Oct 30 16:28:01.796595 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:28:01.796556 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:28:09.395081 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.395049 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6"] Oct 30 16:28:09.395793 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.395274 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" podUID="1b0740dd-2f85-43d5-afa1-5d764ce4ba75" containerName="manager" containerID="cri-o://7364b444fedb5651e6ba5cf933e411ffcb46ed941912b2d25b018d378a6a6282" gracePeriod=2 Oct 30 16:28:09.407012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.406940 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6"] Oct 30 16:28:09.408075 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:28:09.408025 5838 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-fhdv6\" is forbidden: User \"system:node:ip-10-0-137-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-219.ec2.internal' and this object" podUID="1b0740dd-2f85-43d5-afa1-5d764ce4ba75" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:28:09.419383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.419354 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp"] Oct 30 16:28:09.420430 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.419762 5838 cpu_manager.go:414] "containerMap: removing container" podUID="1b0740dd-2f85-43d5-afa1-5d764ce4ba75" containerName="manager" Oct 30 16:28:09.420430 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.419883 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0740dd-2f85-43d5-afa1-5d764ce4ba75" containerName="manager" Oct 30 16:28:09.420430 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.420010 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0740dd-2f85-43d5-afa1-5d764ce4ba75" containerName="manager" Oct 30 16:28:09.423780 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.423761 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp" Oct 30 16:28:09.445703 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.445058 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp"] Oct 30 16:28:09.451637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.451610 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr8kc\" (UniqueName: \"kubernetes.io/projected/f0959ec1-57e5-4472-875b-7ebfec809d5d-kube-api-access-dr8kc\") pod \"limitador-operator-controller-manager-856dd584c5-rpdmp\" (UID: \"f0959ec1-57e5-4472-875b-7ebfec809d5d\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp" Oct 30 16:28:09.478468 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:28:09.478441 5838 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-fhdv6\" is forbidden: User \"system:node:ip-10-0-137-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-219.ec2.internal' and this object" podUID="1b0740dd-2f85-43d5-afa1-5d764ce4ba75" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:28:09.552476 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.552444 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr8kc\" (UniqueName: \"kubernetes.io/projected/f0959ec1-57e5-4472-875b-7ebfec809d5d-kube-api-access-dr8kc\") pod \"limitador-operator-controller-manager-856dd584c5-rpdmp\" (UID: \"f0959ec1-57e5-4472-875b-7ebfec809d5d\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp" Oct 30 16:28:09.575172 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.575135 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr8kc\" (UniqueName: \"kubernetes.io/projected/f0959ec1-57e5-4472-875b-7ebfec809d5d-kube-api-access-dr8kc\") pod \"limitador-operator-controller-manager-856dd584c5-rpdmp\" (UID: \"f0959ec1-57e5-4472-875b-7ebfec809d5d\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp" Oct 30 16:28:09.639855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.639829 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:28:09.652618 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:28:09.652561 5838 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-fhdv6\" is forbidden: User \"system:node:ip-10-0-137-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-219.ec2.internal' and this object" podUID="1b0740dd-2f85-43d5-afa1-5d764ce4ba75" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:28:09.652760 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.652748 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0740dd-2f85-43d5-afa1-5d764ce4ba75-kube-api-access-fbkgd\" (UniqueName: \"kubernetes.io/projected/1b0740dd-2f85-43d5-afa1-5d764ce4ba75-kube-api-access-fbkgd\") pod \"1b0740dd-2f85-43d5-afa1-5d764ce4ba75\" (UID: \"1b0740dd-2f85-43d5-afa1-5d764ce4ba75\") " Oct 30 16:28:09.654862 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.654831 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0740dd-2f85-43d5-afa1-5d764ce4ba75-kube-api-access-fbkgd" pod "1b0740dd-2f85-43d5-afa1-5d764ce4ba75" (UID: "1b0740dd-2f85-43d5-afa1-5d764ce4ba75"). InnerVolumeSpecName "kube-api-access-fbkgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:28:09.753309 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.753278 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbkgd\" (UniqueName: \"kubernetes.io/projected/1b0740dd-2f85-43d5-afa1-5d764ce4ba75-kube-api-access-fbkgd\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:28:09.778831 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:09.778800 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp" Oct 30 16:28:10.080412 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:10.080372 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp"] Oct 30 16:28:10.085037 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:28:10.085009 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0959ec1_57e5_4472_875b_7ebfec809d5d.slice/crio-6afa86fc253f4d8f6e5fe1976a6f1769ba74405eaa45e25f29b8820cce411f73 WatchSource:0}: Error finding container 6afa86fc253f4d8f6e5fe1976a6f1769ba74405eaa45e25f29b8820cce411f73: Status 404 returned error can't find the container with id 6afa86fc253f4d8f6e5fe1976a6f1769ba74405eaa45e25f29b8820cce411f73 Oct 30 16:28:10.287146 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:10.287102 5838 generic.go:356] "Generic (PLEG): container finished" podID="1b0740dd-2f85-43d5-afa1-5d764ce4ba75" containerID="7364b444fedb5651e6ba5cf933e411ffcb46ed941912b2d25b018d378a6a6282" exitCode=0 Oct 30 16:28:10.287328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:10.287172 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:28:10.287328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:10.287206 5838 scope.go:122] "RemoveContainer" containerID="7364b444fedb5651e6ba5cf933e411ffcb46ed941912b2d25b018d378a6a6282" Oct 30 16:28:10.289197 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:10.289167 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp" event={"ID":"f0959ec1-57e5-4472-875b-7ebfec809d5d","Type":"ContainerStarted","Data":"eca03feed254dcb0fe84b75d442f5f8377bb0a00a7e5f2d257194e784bf3c3e5"} Oct 30 16:28:10.289327 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:10.289210 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp" event={"ID":"f0959ec1-57e5-4472-875b-7ebfec809d5d","Type":"ContainerStarted","Data":"6afa86fc253f4d8f6e5fe1976a6f1769ba74405eaa45e25f29b8820cce411f73"} Oct 30 16:28:10.289327 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:10.289256 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp" Oct 30 16:28:10.296692 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:10.296677 5838 scope.go:122] "RemoveContainer" containerID="7364b444fedb5651e6ba5cf933e411ffcb46ed941912b2d25b018d378a6a6282" Oct 30 16:28:10.296953 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:28:10.296936 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7364b444fedb5651e6ba5cf933e411ffcb46ed941912b2d25b018d378a6a6282\": container with ID starting with 7364b444fedb5651e6ba5cf933e411ffcb46ed941912b2d25b018d378a6a6282 not found: ID does not exist" containerID="7364b444fedb5651e6ba5cf933e411ffcb46ed941912b2d25b018d378a6a6282" Oct 30 16:28:10.297012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:10.296961 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7364b444fedb5651e6ba5cf933e411ffcb46ed941912b2d25b018d378a6a6282"} err="failed to get container status \"7364b444fedb5651e6ba5cf933e411ffcb46ed941912b2d25b018d378a6a6282\": rpc error: code = NotFound desc = could not find container \"7364b444fedb5651e6ba5cf933e411ffcb46ed941912b2d25b018d378a6a6282\": container with ID starting with 7364b444fedb5651e6ba5cf933e411ffcb46ed941912b2d25b018d378a6a6282 not found: ID does not exist" Oct 30 16:28:10.300157 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:28:10.300135 5838 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-fhdv6\" is forbidden: User \"system:node:ip-10-0-137-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-219.ec2.internal' and this object" podUID="1b0740dd-2f85-43d5-afa1-5d764ce4ba75" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:28:10.317065 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:28:10.317035 5838 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-fhdv6\" is forbidden: User \"system:node:ip-10-0-137-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-219.ec2.internal' and this object" podUID="1b0740dd-2f85-43d5-afa1-5d764ce4ba75" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fhdv6" Oct 30 16:28:10.363114 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:10.363034 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp" podStartSLOduration=1.36301996 podStartE2EDuration="1.36301996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:10.361576534 +0000 UTC m=+452.110165940" watchObservedRunningTime="2026-10-30 16:28:10.36301996 +0000 UTC m=+452.111609348" Oct 30 16:28:10.393721 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:10.393684 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0740dd-2f85-43d5-afa1-5d764ce4ba75" path="/var/lib/kubelet/pods/1b0740dd-2f85-43d5-afa1-5d764ce4ba75/volumes" Oct 30 16:28:21.298611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:21.298530 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rpdmp" Oct 30 16:28:31.796744 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:28:31.796699 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:28:39.133628 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.133582 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh"] Oct 30 16:28:39.137113 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.137086 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.164829 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.164800 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh"] Oct 30 16:28:39.193370 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.193335 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d96gm\" (UniqueName: \"kubernetes.io/projected/3465e7a5-d6cc-4c8c-9052-29034ba0847f-kube-api-access-d96gm\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.193567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.193377 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.193567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.193461 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.193567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.193488 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.193567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.193510 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.193737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.193606 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.193737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.193701 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.193737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.193732 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.193829 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.193804 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.197049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.197027 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sgq7n\"" Oct 30 16:28:39.294880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.294850 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d96gm\" (UniqueName: \"kubernetes.io/projected/3465e7a5-d6cc-4c8c-9052-29034ba0847f-kube-api-access-d96gm\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295106 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.294885 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295106 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.294914 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295106 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.295037 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295106 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.295081 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.295115 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.295163 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.295203 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.295263 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295518 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.295327 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.295520 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295721 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.295618 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295721 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.295672 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.295849 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.295782 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.298080 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.298052 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.298577 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.298554 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.317776 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.317751 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3465e7a5-d6cc-4c8c-9052-29034ba0847f-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.317898 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.317864 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d96gm\" (UniqueName: \"kubernetes.io/projected/3465e7a5-d6cc-4c8c-9052-29034ba0847f-kube-api-access-d96gm\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-djmdh\" (UID: \"3465e7a5-d6cc-4c8c-9052-29034ba0847f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.450671 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.450576 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:39.764089 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.764053 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh"] Oct 30 16:28:39.767815 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:28:39.767792 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3465e7a5_d6cc_4c8c_9052_29034ba0847f.slice/crio-a747147557b91c7d9efa8bca9698c4b083ead2f12443cf74aadc6f75ffbce5d9 WatchSource:0}: Error finding container a747147557b91c7d9efa8bca9698c4b083ead2f12443cf74aadc6f75ffbce5d9: Status 404 returned error can't find the container with id a747147557b91c7d9efa8bca9698c4b083ead2f12443cf74aadc6f75ffbce5d9 Oct 30 16:28:39.769884 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.769856 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:28:39.769958 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.769918 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:28:39.769958 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:39.769947 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:28:40.444376 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:40.444284 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" event={"ID":"3465e7a5-d6cc-4c8c-9052-29034ba0847f","Type":"ContainerStarted","Data":"5fa68db8e8cbd6e1a24adac8af0ca91d33bad388fa5a7cd828ec6617fd3cf0ce"} Oct 30 16:28:40.444376 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:40.444346 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" event={"ID":"3465e7a5-d6cc-4c8c-9052-29034ba0847f","Type":"ContainerStarted","Data":"a747147557b91c7d9efa8bca9698c4b083ead2f12443cf74aadc6f75ffbce5d9"} Oct 30 16:28:40.451726 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:40.451703 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:40.456448 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:40.456425 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:40.501066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:40.501012 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" podStartSLOduration=1.500998319 podStartE2EDuration="1.500998319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:40.499595959 +0000 UTC m=+482.248185366" watchObservedRunningTime="2026-10-30 16:28:40.500998319 +0000 UTC m=+482.249587703" Oct 30 16:28:41.449751 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:41.449719 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:41.451069 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:41.451045 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:28:54.270231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.270187 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:28:54.273101 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.273076 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" Oct 30 16:28:54.289799 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.289776 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 16:28:54.299047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.299015 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j4w26\"" Oct 30 16:28:54.301014 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.300990 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:28:54.345275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.345204 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:28:54.432740 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.432698 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86kn4\" (UniqueName: \"kubernetes.io/projected/89846f13-eb01-4f1e-9abb-2d5ad16c2265-kube-api-access-86kn4\") pod \"limitador-limitador-7c9b87d574-zjffw\" (UID: \"89846f13-eb01-4f1e-9abb-2d5ad16c2265\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" Oct 30 16:28:54.432929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.432870 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/89846f13-eb01-4f1e-9abb-2d5ad16c2265-config-file\") pod \"limitador-limitador-7c9b87d574-zjffw\" (UID: \"89846f13-eb01-4f1e-9abb-2d5ad16c2265\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" Oct 30 16:28:54.534113 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.534037 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/89846f13-eb01-4f1e-9abb-2d5ad16c2265-config-file\") pod \"limitador-limitador-7c9b87d574-zjffw\" (UID: \"89846f13-eb01-4f1e-9abb-2d5ad16c2265\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" Oct 30 16:28:54.534113 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.534081 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86kn4\" (UniqueName: \"kubernetes.io/projected/89846f13-eb01-4f1e-9abb-2d5ad16c2265-kube-api-access-86kn4\") pod \"limitador-limitador-7c9b87d574-zjffw\" (UID: \"89846f13-eb01-4f1e-9abb-2d5ad16c2265\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" Oct 30 16:28:54.534731 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.534701 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/89846f13-eb01-4f1e-9abb-2d5ad16c2265-config-file\") pod \"limitador-limitador-7c9b87d574-zjffw\" (UID: \"89846f13-eb01-4f1e-9abb-2d5ad16c2265\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" Oct 30 16:28:54.557267 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.557236 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86kn4\" (UniqueName: \"kubernetes.io/projected/89846f13-eb01-4f1e-9abb-2d5ad16c2265-kube-api-access-86kn4\") pod \"limitador-limitador-7c9b87d574-zjffw\" (UID: \"89846f13-eb01-4f1e-9abb-2d5ad16c2265\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" Oct 30 16:28:54.585725 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.585696 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" Oct 30 16:28:54.882303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:54.882211 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:28:54.886377 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:28:54.886352 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89846f13_eb01_4f1e_9abb_2d5ad16c2265.slice/crio-53d667145ff3565465b8e25ef46a3d28944f29f50c52773820e3ec2591a894a2 WatchSource:0}: Error finding container 53d667145ff3565465b8e25ef46a3d28944f29f50c52773820e3ec2591a894a2: Status 404 returned error can't find the container with id 53d667145ff3565465b8e25ef46a3d28944f29f50c52773820e3ec2591a894a2 Oct 30 16:28:55.518338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:55.518291 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" event={"ID":"89846f13-eb01-4f1e-9abb-2d5ad16c2265","Type":"ContainerStarted","Data":"53d667145ff3565465b8e25ef46a3d28944f29f50c52773820e3ec2591a894a2"} Oct 30 16:28:56.524567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:56.524527 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" event={"ID":"89846f13-eb01-4f1e-9abb-2d5ad16c2265","Type":"ContainerStarted","Data":"7cd7422d9f84aa542307892a806177992364d8ea1cb0b654b7d2dd55e01e3634"} Oct 30 16:28:56.524956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:56.524599 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" Oct 30 16:28:56.565495 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:28:56.565435 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" podStartSLOduration=1.788843751 podStartE2EDuration="2.565422737s" totalImagesPullingTime="776.578986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:56.563476604 +0000 UTC m=+498.312066016" watchObservedRunningTime="2026-10-30 16:28:56.565422737 +0000 UTC m=+498.314012123" Oct 30 16:29:01.795896 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:29:01.795859 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:29:07.530791 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:29:07.530751 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-zjffw" Oct 30 16:29:31.796633 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:29:31.796594 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:30:01.795392 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:30:01.795314 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:30:16.578209 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:16.578156 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:30:25.994653 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:25.994617 5838 generic.go:356] "Generic (PLEG): container finished" podID="3465e7a5-d6cc-4c8c-9052-29034ba0847f" containerID="5fa68db8e8cbd6e1a24adac8af0ca91d33bad388fa5a7cd828ec6617fd3cf0ce" exitCode=0 Oct 30 16:30:25.995100 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:25.994671 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" event={"ID":"3465e7a5-d6cc-4c8c-9052-29034ba0847f","Type":"ContainerDied","Data":"5fa68db8e8cbd6e1a24adac8af0ca91d33bad388fa5a7cd828ec6617fd3cf0ce"} Oct 30 16:30:25.995100 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:25.995043 5838 scope.go:122] "RemoveContainer" containerID="5fa68db8e8cbd6e1a24adac8af0ca91d33bad388fa5a7cd828ec6617fd3cf0ce" Oct 30 16:30:25.996131 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:25.996101 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:30:25.996200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:25.996173 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:30:25.996235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:25.996203 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:30:26.450314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:26.450220 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:30:27.001740 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:27.001706 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" event={"ID":"3465e7a5-d6cc-4c8c-9052-29034ba0847f","Type":"ContainerStarted","Data":"e90dfcf83c31ca46d67cc318114d4c427b7fcd1eadbe1d4a61d3fa53f35bc514"} Oct 30 16:30:27.451771 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:27.451673 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:30:27.452821 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:27.452794 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 16:30:27.452947 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:27.452878 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" podUID="3465e7a5-d6cc-4c8c-9052-29034ba0847f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 16:30:28.451634 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:28.451598 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 16:30:28.452064 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:28.451655 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" podUID="3465e7a5-d6cc-4c8c-9052-29034ba0847f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 16:30:29.451292 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:29.451254 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 16:30:29.451485 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:29.451325 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" podUID="3465e7a5-d6cc-4c8c-9052-29034ba0847f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 16:30:30.451153 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:30.451115 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 16:30:30.451553 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:30.451186 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" podUID="3465e7a5-d6cc-4c8c-9052-29034ba0847f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 16:30:31.451061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:31.451022 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 16:30:31.451246 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:31.451079 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" podUID="3465e7a5-d6cc-4c8c-9052-29034ba0847f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 16:30:31.797448 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:30:31.797410 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:30:32.455199 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:32.455166 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:30:32.455598 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:32.455548 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:30:32.456365 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:32.456345 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" Oct 30 16:30:38.448037 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:38.448009 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:30:38.448494 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:30:38.448056 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:31:01.796797 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:31:01.796761 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:31:22.030178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:22.030078 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:31:24.458653 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:24.458610 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:31:27.920048 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:27.920008 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:31:29.273501 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.273469 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn"] Oct 30 16:31:29.277136 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.277113 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.303164 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.303116 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 16:31:29.303164 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.303129 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fv4q6\"" Oct 30 16:31:29.303164 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.303134 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 16:31:29.303458 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.303297 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 16:31:29.305770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.305741 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn"] Oct 30 16:31:29.385062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.385025 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a60bebc-f990-43e5-b655-458e6bf057c3-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.385251 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.385084 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a60bebc-f990-43e5-b655-458e6bf057c3-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.385251 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.385202 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a60bebc-f990-43e5-b655-458e6bf057c3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.385251 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.385232 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26b9b\" (UniqueName: \"kubernetes.io/projected/5a60bebc-f990-43e5-b655-458e6bf057c3-kube-api-access-26b9b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.385388 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.385267 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a60bebc-f990-43e5-b655-458e6bf057c3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.385388 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.385345 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a60bebc-f990-43e5-b655-458e6bf057c3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.486747 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.486695 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a60bebc-f990-43e5-b655-458e6bf057c3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.486747 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.486753 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a60bebc-f990-43e5-b655-458e6bf057c3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.487047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.486780 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a60bebc-f990-43e5-b655-458e6bf057c3-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.487047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.486814 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a60bebc-f990-43e5-b655-458e6bf057c3-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.487047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.486851 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a60bebc-f990-43e5-b655-458e6bf057c3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.487047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.486868 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26b9b\" (UniqueName: \"kubernetes.io/projected/5a60bebc-f990-43e5-b655-458e6bf057c3-kube-api-access-26b9b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.487328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.487302 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a60bebc-f990-43e5-b655-458e6bf057c3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.487402 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.487346 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a60bebc-f990-43e5-b655-458e6bf057c3-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.487464 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.487414 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a60bebc-f990-43e5-b655-458e6bf057c3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.489231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.489211 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a60bebc-f990-43e5-b655-458e6bf057c3-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.489403 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.489387 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a60bebc-f990-43e5-b655-458e6bf057c3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.510748 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.510712 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26b9b\" (UniqueName: \"kubernetes.io/projected/5a60bebc-f990-43e5-b655-458e6bf057c3-kube-api-access-26b9b\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn\" (UID: \"5a60bebc-f990-43e5-b655-458e6bf057c3\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.590592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.590497 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:29.892405 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:29.892298 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn"] Oct 30 16:31:29.897525 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:31:29.897491 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a60bebc_f990_43e5_b655_458e6bf057c3.slice/crio-970f9665602a497fc560858600e8b08b489ef1572fa934a1f578e4f760dc22d6 WatchSource:0}: Error finding container 970f9665602a497fc560858600e8b08b489ef1572fa934a1f578e4f760dc22d6: Status 404 returned error can't find the container with id 970f9665602a497fc560858600e8b08b489ef1572fa934a1f578e4f760dc22d6 Oct 30 16:31:30.328535 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:30.328487 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" event={"ID":"5a60bebc-f990-43e5-b655-458e6bf057c3","Type":"ContainerStarted","Data":"970f9665602a497fc560858600e8b08b489ef1572fa934a1f578e4f760dc22d6"} Oct 30 16:31:31.335940 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.335890 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" event={"ID":"5a60bebc-f990-43e5-b655-458e6bf057c3","Type":"ContainerStarted","Data":"d295decadf2714884fc8d0ef4f3fde6cb42c0b98f1bd010e17f8dc120535d6c8"} Oct 30 16:31:31.378798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.378720 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" podStartSLOduration=1.32123558 podStartE2EDuration="2.378700694s" totalImagesPullingTime="1.057465114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:31.377091629 +0000 UTC m=+653.125681016" watchObservedRunningTime="2026-10-30 16:31:31.378700694 +0000 UTC m=+653.127290083" Oct 30 16:31:31.591015 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.590918 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:31.654788 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.654748 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq"] Oct 30 16:31:31.662319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.662283 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.678708 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.678675 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 16:31:31.682403 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.682373 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq"] Oct 30 16:31:31.707840 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.707807 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.708047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.707999 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.708047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.708027 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.708162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.708049 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff9gp\" (UniqueName: \"kubernetes.io/projected/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-kube-api-access-ff9gp\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.708162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.708118 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.708162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.708140 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.798527 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:31:31.798488 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:31:31.809549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.809521 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.809678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.809564 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.809678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.809582 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.809678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.809597 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff9gp\" (UniqueName: \"kubernetes.io/projected/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-kube-api-access-ff9gp\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.809678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.809615 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.809678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.809636 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.810002 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.809954 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.810078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.810030 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.810136 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.810116 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.811956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.811933 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.812307 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.812291 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.839440 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.839403 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff9gp\" (UniqueName: \"kubernetes.io/projected/36e0fb5d-ff56-4fd3-a4a6-683306972ef7-kube-api-access-ff9gp\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq\" (UID: \"36e0fb5d-ff56-4fd3-a4a6-683306972ef7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:31.985320 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:31.985214 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:32.284403 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:32.284285 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq"] Oct 30 16:31:32.289180 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:31:32.289150 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36e0fb5d_ff56_4fd3_a4a6_683306972ef7.slice/crio-4a931ea881f51dbaa7608011e1d88ee3a9db0a8495e844bedde48af77ff74f1f WatchSource:0}: Error finding container 4a931ea881f51dbaa7608011e1d88ee3a9db0a8495e844bedde48af77ff74f1f: Status 404 returned error can't find the container with id 4a931ea881f51dbaa7608011e1d88ee3a9db0a8495e844bedde48af77ff74f1f Oct 30 16:31:32.341779 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:32.341741 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" event={"ID":"36e0fb5d-ff56-4fd3-a4a6-683306972ef7","Type":"ContainerStarted","Data":"4a931ea881f51dbaa7608011e1d88ee3a9db0a8495e844bedde48af77ff74f1f"} Oct 30 16:31:33.006572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:33.006530 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:31:33.349343 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:33.349254 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" event={"ID":"36e0fb5d-ff56-4fd3-a4a6-683306972ef7","Type":"ContainerStarted","Data":"3fcb43e5cb2b8d5070447687585d93c02730eb64794c0b1fc28e9bd24dd7157d"} Oct 30 16:31:33.395391 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:33.395325 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podStartSLOduration=2.16590754 podStartE2EDuration="2.395300742s" totalImagesPullingTime="229.393202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:33.392917501 +0000 UTC m=+655.141506887" watchObservedRunningTime="2026-10-30 16:31:33.395300742 +0000 UTC m=+655.143890129" Oct 30 16:31:33.985710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:33.985663 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:39.591441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:39.591406 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:39.603827 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:39.603799 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:40.410816 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:40.410731 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn" Oct 30 16:31:40.511043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:40.511008 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:31:41.986251 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:41.986207 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:41.998443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:41.998414 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:42.421158 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:42.421075 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:31:50.326428 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.326384 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-94sgf"] Oct 30 16:31:50.330680 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.330660 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" Oct 30 16:31:50.357076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.357042 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 16:31:50.357246 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.357076 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 16:31:50.357246 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.357199 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-pljkn\"" Oct 30 16:31:50.359636 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.359611 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-94sgf"] Oct 30 16:31:50.482409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.482377 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-metrics-certs\") pod \"ai-gateway-controller-84dff4756b-94sgf\" (UID: \"310ab042-f53a-48f1-8e4a-e8d4e2feb58c\") " pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" Oct 30 16:31:50.482595 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.482417 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tljn\" (UniqueName: \"kubernetes.io/projected/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-kube-api-access-8tljn\") pod \"ai-gateway-controller-84dff4756b-94sgf\" (UID: \"310ab042-f53a-48f1-8e4a-e8d4e2feb58c\") " pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" Oct 30 16:31:50.584689 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.584596 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-metrics-certs\") pod \"ai-gateway-controller-84dff4756b-94sgf\" (UID: \"310ab042-f53a-48f1-8e4a-e8d4e2feb58c\") " pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" Oct 30 16:31:50.584923 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.584900 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tljn\" (UniqueName: \"kubernetes.io/projected/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-kube-api-access-8tljn\") pod \"ai-gateway-controller-84dff4756b-94sgf\" (UID: \"310ab042-f53a-48f1-8e4a-e8d4e2feb58c\") " pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" Oct 30 16:31:50.596123 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.596085 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-metrics-certs\") pod \"ai-gateway-controller-84dff4756b-94sgf\" (UID: \"310ab042-f53a-48f1-8e4a-e8d4e2feb58c\") " pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" Oct 30 16:31:50.613880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.613816 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tljn\" (UniqueName: \"kubernetes.io/projected/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-kube-api-access-8tljn\") pod \"ai-gateway-controller-84dff4756b-94sgf\" (UID: \"310ab042-f53a-48f1-8e4a-e8d4e2feb58c\") " pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" Oct 30 16:31:50.646111 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.646061 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" Oct 30 16:31:50.956807 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:50.956694 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-94sgf"] Oct 30 16:31:50.960437 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:31:50.960407 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310ab042_f53a_48f1_8e4a_e8d4e2feb58c.slice/crio-9e8b17b0d7d94a90e59151dc393ec11e8b649a8bea3b4fa5e6935a42c14bd312 WatchSource:0}: Error finding container 9e8b17b0d7d94a90e59151dc393ec11e8b649a8bea3b4fa5e6935a42c14bd312: Status 404 returned error can't find the container with id 9e8b17b0d7d94a90e59151dc393ec11e8b649a8bea3b4fa5e6935a42c14bd312 Oct 30 16:31:51.454202 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:51.454162 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" event={"ID":"310ab042-f53a-48f1-8e4a-e8d4e2feb58c","Type":"ContainerStarted","Data":"9e8b17b0d7d94a90e59151dc393ec11e8b649a8bea3b4fa5e6935a42c14bd312"} Oct 30 16:31:53.466380 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:53.466344 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" event={"ID":"310ab042-f53a-48f1-8e4a-e8d4e2feb58c","Type":"ContainerStarted","Data":"b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036"} Oct 30 16:31:53.466750 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:53.466406 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" Oct 30 16:31:53.506272 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:53.506215 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" podStartSLOduration=1.171385359 podStartE2EDuration="3.506200412s" totalImagesPullingTime="2.334815053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:53.505753572 +0000 UTC m=+675.254342982" watchObservedRunningTime="2026-10-30 16:31:53.506200412 +0000 UTC m=+675.254789799" Oct 30 16:31:55.111903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.111867 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns"] Oct 30 16:31:55.115626 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.115603 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.146383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.146349 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns"] Oct 30 16:31:55.226913 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.226874 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.227115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.226965 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.227115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.227014 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.227115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.227039 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.227115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.227063 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.227323 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.227130 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.227323 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.227168 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.227323 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.227209 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrs9h\" (UniqueName: \"kubernetes.io/projected/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-kube-api-access-wrs9h\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.227323 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.227248 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.328579 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.328540 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.328786 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.328589 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.328786 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.328635 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.328786 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.328658 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.328786 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.328692 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrs9h\" (UniqueName: \"kubernetes.io/projected/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-kube-api-access-wrs9h\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.329030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.328790 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.329030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.328901 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.329030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.328961 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.329206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.329030 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.329206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.329075 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.329206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.329104 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.329206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.329177 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.329418 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.329324 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.329471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.329434 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.331334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.331311 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.331565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.331545 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.353646 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.353613 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrs9h\" (UniqueName: \"kubernetes.io/projected/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-kube-api-access-wrs9h\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.354047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.353961 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fecda1cb-458a-4c3e-b48f-a0e6d82c1995-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-x6sns\" (UID: \"fecda1cb-458a-4c3e-b48f-a0e6d82c1995\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.429200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.429109 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:55.609277 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.609243 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln"] Oct 30 16:31:55.613280 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.613260 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" Oct 30 16:31:55.631630 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.631603 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 16:31:55.631789 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.631662 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6xjww\"" Oct 30 16:31:55.633901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.633872 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln"] Oct 30 16:31:55.679139 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.679098 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-9hs5j"] Oct 30 16:31:55.687012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.684902 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" Oct 30 16:31:55.702078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.702045 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-9hs5j"] Oct 30 16:31:55.733039 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.733006 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aeeae52-0ed0-4b70-9da0-640a6dfac794-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-sv5ln\" (UID: \"8aeeae52-0ed0-4b70-9da0-640a6dfac794\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" Oct 30 16:31:55.733235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.733045 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmqkn\" (UniqueName: \"kubernetes.io/projected/8aeeae52-0ed0-4b70-9da0-640a6dfac794-kube-api-access-tmqkn\") pod \"payload-pre-processing-7b98bb678c-sv5ln\" (UID: \"8aeeae52-0ed0-4b70-9da0-640a6dfac794\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" Oct 30 16:31:55.764420 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.764388 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns"] Oct 30 16:31:55.768603 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:31:55.768573 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecda1cb_458a_4c3e_b48f_a0e6d82c1995.slice/crio-b5b1d986d475a88a61dcf867e2321a19c5ec059dbfb445ea8ee592cb0f1ef0fd WatchSource:0}: Error finding container b5b1d986d475a88a61dcf867e2321a19c5ec059dbfb445ea8ee592cb0f1ef0fd: Status 404 returned error can't find the container with id b5b1d986d475a88a61dcf867e2321a19c5ec059dbfb445ea8ee592cb0f1ef0fd Oct 30 16:31:55.770663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.770633 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:31:55.770724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.770704 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:31:55.770769 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.770739 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:31:55.833624 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.833598 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aeeae52-0ed0-4b70-9da0-640a6dfac794-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-sv5ln\" (UID: \"8aeeae52-0ed0-4b70-9da0-640a6dfac794\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" Oct 30 16:31:55.833766 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.833633 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmqkn\" (UniqueName: \"kubernetes.io/projected/8aeeae52-0ed0-4b70-9da0-640a6dfac794-kube-api-access-tmqkn\") pod \"payload-pre-processing-7b98bb678c-sv5ln\" (UID: \"8aeeae52-0ed0-4b70-9da0-640a6dfac794\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" Oct 30 16:31:55.833766 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.833684 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f776e7-ae4d-4ccc-ab01-d107e80df07e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-9hs5j\" (UID: \"72f776e7-ae4d-4ccc-ab01-d107e80df07e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" Oct 30 16:31:55.833766 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.833731 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8h2\" (UniqueName: \"kubernetes.io/projected/72f776e7-ae4d-4ccc-ab01-d107e80df07e-kube-api-access-9t8h2\") pod \"payload-processing-55fbbbf9dc-9hs5j\" (UID: \"72f776e7-ae4d-4ccc-ab01-d107e80df07e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" Oct 30 16:31:55.834359 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.834337 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aeeae52-0ed0-4b70-9da0-640a6dfac794-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-sv5ln\" (UID: \"8aeeae52-0ed0-4b70-9da0-640a6dfac794\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" Oct 30 16:31:55.861294 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.861261 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmqkn\" (UniqueName: \"kubernetes.io/projected/8aeeae52-0ed0-4b70-9da0-640a6dfac794-kube-api-access-tmqkn\") pod \"payload-pre-processing-7b98bb678c-sv5ln\" (UID: \"8aeeae52-0ed0-4b70-9da0-640a6dfac794\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" Oct 30 16:31:55.926889 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.926857 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" Oct 30 16:31:55.934305 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.934273 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8h2\" (UniqueName: \"kubernetes.io/projected/72f776e7-ae4d-4ccc-ab01-d107e80df07e-kube-api-access-9t8h2\") pod \"payload-processing-55fbbbf9dc-9hs5j\" (UID: \"72f776e7-ae4d-4ccc-ab01-d107e80df07e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" Oct 30 16:31:55.934407 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.934363 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f776e7-ae4d-4ccc-ab01-d107e80df07e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-9hs5j\" (UID: \"72f776e7-ae4d-4ccc-ab01-d107e80df07e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" Oct 30 16:31:55.935020 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.935000 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f776e7-ae4d-4ccc-ab01-d107e80df07e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-9hs5j\" (UID: \"72f776e7-ae4d-4ccc-ab01-d107e80df07e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" Oct 30 16:31:55.957711 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.957673 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8h2\" (UniqueName: \"kubernetes.io/projected/72f776e7-ae4d-4ccc-ab01-d107e80df07e-kube-api-access-9t8h2\") pod \"payload-processing-55fbbbf9dc-9hs5j\" (UID: \"72f776e7-ae4d-4ccc-ab01-d107e80df07e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" Oct 30 16:31:55.999450 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:55.999405 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" Oct 30 16:31:56.239945 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:56.239905 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln"] Oct 30 16:31:56.243619 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:31:56.243595 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aeeae52_0ed0_4b70_9da0_640a6dfac794.slice/crio-66734b7858ed76d32c4db40ad381a202a12c94c4e978a86b97241464d0ca000b WatchSource:0}: Error finding container 66734b7858ed76d32c4db40ad381a202a12c94c4e978a86b97241464d0ca000b: Status 404 returned error can't find the container with id 66734b7858ed76d32c4db40ad381a202a12c94c4e978a86b97241464d0ca000b Oct 30 16:31:56.307367 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:56.307196 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-9hs5j"] Oct 30 16:31:56.312187 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:31:56.312149 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f776e7_ae4d_4ccc_ab01_d107e80df07e.slice/crio-ad903341654e3eb08257c2b6e2706921b484c485476744d9cfd5ce4c05361e2f WatchSource:0}: Error finding container ad903341654e3eb08257c2b6e2706921b484c485476744d9cfd5ce4c05361e2f: Status 404 returned error can't find the container with id ad903341654e3eb08257c2b6e2706921b484c485476744d9cfd5ce4c05361e2f Oct 30 16:31:56.482389 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:56.482345 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" event={"ID":"fecda1cb-458a-4c3e-b48f-a0e6d82c1995","Type":"ContainerStarted","Data":"84c84d6d2140a4ba17c1c0b56ce40f355cbc58320324380cb849fbbd1cff1844"} Oct 30 16:31:56.482560 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:56.482403 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" event={"ID":"fecda1cb-458a-4c3e-b48f-a0e6d82c1995","Type":"ContainerStarted","Data":"b5b1d986d475a88a61dcf867e2321a19c5ec059dbfb445ea8ee592cb0f1ef0fd"} Oct 30 16:31:56.483500 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:56.483476 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" event={"ID":"72f776e7-ae4d-4ccc-ab01-d107e80df07e","Type":"ContainerStarted","Data":"ad903341654e3eb08257c2b6e2706921b484c485476744d9cfd5ce4c05361e2f"} Oct 30 16:31:56.484576 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:56.484557 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" event={"ID":"8aeeae52-0ed0-4b70-9da0-640a6dfac794","Type":"ContainerStarted","Data":"66734b7858ed76d32c4db40ad381a202a12c94c4e978a86b97241464d0ca000b"} Oct 30 16:31:56.541051 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:56.540923 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" podStartSLOduration=1.540908821 podStartE2EDuration="1.540908821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:56.53563716 +0000 UTC m=+678.284226555" watchObservedRunningTime="2026-10-30 16:31:56.540908821 +0000 UTC m=+678.289498275" Oct 30 16:31:57.441446 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:57.441162 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:31:57.447846 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:57.446183 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:31:57.447846 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:57.446247 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" podUID="fecda1cb-458a-4c3e-b48f-a0e6d82c1995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:31:58.440569 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:58.440530 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:31:58.440770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:58.440615 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" podUID="fecda1cb-458a-4c3e-b48f-a0e6d82c1995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:31:59.430994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:59.430435 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:31:59.430994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:59.430497 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" podUID="fecda1cb-458a-4c3e-b48f-a0e6d82c1995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:31:59.509634 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:59.509533 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" event={"ID":"72f776e7-ae4d-4ccc-ab01-d107e80df07e","Type":"ContainerStarted","Data":"dbf281ee25fd05df59356321fafffabe60a485eb1fd3293b7ef10bdff048b701"} Oct 30 16:31:59.509802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:59.509645 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" Oct 30 16:31:59.511238 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:59.511212 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" event={"ID":"8aeeae52-0ed0-4b70-9da0-640a6dfac794","Type":"ContainerStarted","Data":"8b3d095a822733a64ae0299f9b141f8fde22f636d7281ca1404495f504ffc8c9"} Oct 30 16:31:59.511360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:59.511298 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" Oct 30 16:31:59.562937 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:59.562872 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" podStartSLOduration=1.63268002 podStartE2EDuration="4.56285614s" totalImagesPullingTime="2.93017612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:59.56166071 +0000 UTC m=+681.310250117" watchObservedRunningTime="2026-10-30 16:31:59.56285614 +0000 UTC m=+681.311445529" Oct 30 16:31:59.602124 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:31:59.602070 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" podStartSLOduration=1.539626765 podStartE2EDuration="4.602057439s" totalImagesPullingTime="3.062430674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:59.601417407 +0000 UTC m=+681.350006805" watchObservedRunningTime="2026-10-30 16:31:59.602057439 +0000 UTC m=+681.350646805" Oct 30 16:32:00.438999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:00.434453 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:32:00.438999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:00.434522 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" podUID="fecda1cb-458a-4c3e-b48f-a0e6d82c1995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:32:01.435994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:01.435427 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:32:01.435994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:01.435515 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" podUID="fecda1cb-458a-4c3e-b48f-a0e6d82c1995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:32:01.798295 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:32:01.798245 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:32:02.429686 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:02.429650 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 16:32:02.429899 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:02.429708 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" podUID="fecda1cb-458a-4c3e-b48f-a0e6d82c1995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 16:32:04.120125 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.120088 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:32:04.121049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.121017 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:32:04.122144 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.122108 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" Oct 30 16:32:04.474200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.474107 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" Oct 30 16:32:04.760932 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.760881 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-67b6f84467-248zx"] Oct 30 16:32:04.766777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.766741 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" Oct 30 16:32:04.784910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.784876 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-67b6f84467-248zx"] Oct 30 16:32:04.824076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.824041 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4943e10f-4269-4eba-a25d-4996fa2fb20a-metrics-certs\") pod \"ai-gateway-controller-67b6f84467-248zx\" (UID: \"4943e10f-4269-4eba-a25d-4996fa2fb20a\") " pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" Oct 30 16:32:04.824272 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.824089 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6j9l\" (UniqueName: \"kubernetes.io/projected/4943e10f-4269-4eba-a25d-4996fa2fb20a-kube-api-access-m6j9l\") pod \"ai-gateway-controller-67b6f84467-248zx\" (UID: \"4943e10f-4269-4eba-a25d-4996fa2fb20a\") " pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" Oct 30 16:32:04.924801 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.924762 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4943e10f-4269-4eba-a25d-4996fa2fb20a-metrics-certs\") pod \"ai-gateway-controller-67b6f84467-248zx\" (UID: \"4943e10f-4269-4eba-a25d-4996fa2fb20a\") " pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" Oct 30 16:32:04.925055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.924821 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6j9l\" (UniqueName: \"kubernetes.io/projected/4943e10f-4269-4eba-a25d-4996fa2fb20a-kube-api-access-m6j9l\") pod \"ai-gateway-controller-67b6f84467-248zx\" (UID: \"4943e10f-4269-4eba-a25d-4996fa2fb20a\") " pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" Oct 30 16:32:04.927536 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.927500 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4943e10f-4269-4eba-a25d-4996fa2fb20a-metrics-certs\") pod \"ai-gateway-controller-67b6f84467-248zx\" (UID: \"4943e10f-4269-4eba-a25d-4996fa2fb20a\") " pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" Oct 30 16:32:04.948182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:04.948149 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6j9l\" (UniqueName: \"kubernetes.io/projected/4943e10f-4269-4eba-a25d-4996fa2fb20a-kube-api-access-m6j9l\") pod \"ai-gateway-controller-67b6f84467-248zx\" (UID: \"4943e10f-4269-4eba-a25d-4996fa2fb20a\") " pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" Oct 30 16:32:05.079457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:05.079352 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" Oct 30 16:32:05.382845 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:05.382723 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-67b6f84467-248zx"] Oct 30 16:32:05.387423 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:05.387392 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4943e10f_4269_4eba_a25d_4996fa2fb20a.slice/crio-3d0b507b60339db8427ff7b151cf3f563096c7064b6038c87e9c90eb1882e267 WatchSource:0}: Error finding container 3d0b507b60339db8427ff7b151cf3f563096c7064b6038c87e9c90eb1882e267: Status 404 returned error can't find the container with id 3d0b507b60339db8427ff7b151cf3f563096c7064b6038c87e9c90eb1882e267 Oct 30 16:32:05.565357 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:05.565316 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" event={"ID":"4943e10f-4269-4eba-a25d-4996fa2fb20a","Type":"ContainerStarted","Data":"3d0b507b60339db8427ff7b151cf3f563096c7064b6038c87e9c90eb1882e267"} Oct 30 16:32:06.572436 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:06.572399 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" event={"ID":"4943e10f-4269-4eba-a25d-4996fa2fb20a","Type":"ContainerStarted","Data":"47992b59c4568609518d7233d6825d427ebf464edeeaf3afb85df1e4760a71e6"} Oct 30 16:32:06.572832 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:06.572464 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" Oct 30 16:32:06.625686 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:06.625625 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" podStartSLOduration=2.352041186 podStartE2EDuration="2.62560438s" totalImagesPullingTime="273.563194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:06.624441327 +0000 UTC m=+688.373030729" watchObservedRunningTime="2026-10-30 16:32:06.62560438 +0000 UTC m=+688.374193768" Oct 30 16:32:10.151777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.151742 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9"] Oct 30 16:32:10.154903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.154882 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.171094 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.171057 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.171094 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.171094 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006df6fa-0ef9-4007-869e-0a251d92c372-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.171321 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.171113 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.171321 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.171227 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.171321 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.171310 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2wm\" (UniqueName: \"kubernetes.io/projected/006df6fa-0ef9-4007-869e-0a251d92c372-kube-api-access-6h2wm\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.171498 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.171348 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.171498 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.171374 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006df6fa-0ef9-4007-869e-0a251d92c372-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.171498 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.171422 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.171498 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.171444 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006df6fa-0ef9-4007-869e-0a251d92c372-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.183792 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.183748 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9"] Oct 30 16:32:10.272001 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.271934 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272009 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006df6fa-0ef9-4007-869e-0a251d92c372-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272032 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272062 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272113 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2wm\" (UniqueName: \"kubernetes.io/projected/006df6fa-0ef9-4007-869e-0a251d92c372-kube-api-access-6h2wm\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272148 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272166 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006df6fa-0ef9-4007-869e-0a251d92c372-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272183 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272564 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272210 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006df6fa-0ef9-4007-869e-0a251d92c372-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272564 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272390 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-workload-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272564 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272432 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-istio-data\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272564 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272469 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-credential-socket\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272564 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272505 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-workload-certs\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.272862 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.272843 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006df6fa-0ef9-4007-869e-0a251d92c372-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.274494 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.274472 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006df6fa-0ef9-4007-869e-0a251d92c372-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.274681 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.274662 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006df6fa-0ef9-4007-869e-0a251d92c372-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.294630 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.294592 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006df6fa-0ef9-4007-869e-0a251d92c372-istio-token\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.294799 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.294776 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2wm\" (UniqueName: \"kubernetes.io/projected/006df6fa-0ef9-4007-869e-0a251d92c372-kube-api-access-6h2wm\") pod \"maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9\" (UID: \"006df6fa-0ef9-4007-869e-0a251d92c372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.470372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.470269 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:10.533220 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.532261 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sv5ln" Oct 30 16:32:10.542512 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.542487 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-9hs5j" Oct 30 16:32:10.800653 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.800616 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9"] Oct 30 16:32:10.805049 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:10.805021 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006df6fa_0ef9_4007_869e_0a251d92c372.slice/crio-55a0e2bed54ed03fd5e79647fb7b563ed7489b20102fee5634143c87f16ed160 WatchSource:0}: Error finding container 55a0e2bed54ed03fd5e79647fb7b563ed7489b20102fee5634143c87f16ed160: Status 404 returned error can't find the container with id 55a0e2bed54ed03fd5e79647fb7b563ed7489b20102fee5634143c87f16ed160 Oct 30 16:32:10.807190 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.807159 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:10.807290 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.807239 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:10.807290 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:10.807282 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:11.601594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:11.601556 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" event={"ID":"006df6fa-0ef9-4007-869e-0a251d92c372","Type":"ContainerStarted","Data":"15af43e90f2cd8ead8b035c26c3cb92539a3f8e103f6d96b470cb143860d8fd9"} Oct 30 16:32:11.602074 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:11.601607 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" event={"ID":"006df6fa-0ef9-4007-869e-0a251d92c372","Type":"ContainerStarted","Data":"55a0e2bed54ed03fd5e79647fb7b563ed7489b20102fee5634143c87f16ed160"} Oct 30 16:32:11.684398 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:11.684337 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" podStartSLOduration=1.684323194 podStartE2EDuration="1.684323194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:11.672140188 +0000 UTC m=+693.420729596" watchObservedRunningTime="2026-10-30 16:32:11.684323194 +0000 UTC m=+693.432912580" Oct 30 16:32:12.471207 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:12.471175 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:12.472709 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:12.472686 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 16:32:12.472765 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:12.472733 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" podUID="006df6fa-0ef9-4007-869e-0a251d92c372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 16:32:13.471139 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:13.471103 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 16:32:13.471554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:13.471159 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" podUID="006df6fa-0ef9-4007-869e-0a251d92c372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 16:32:14.470809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:14.470772 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 16:32:14.471015 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:14.470828 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" podUID="006df6fa-0ef9-4007-869e-0a251d92c372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 16:32:15.471369 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:15.471327 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 16:32:15.471867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:15.471384 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" podUID="006df6fa-0ef9-4007-869e-0a251d92c372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 16:32:16.471321 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:16.471278 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 16:32:16.471500 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:16.471339 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" podUID="006df6fa-0ef9-4007-869e-0a251d92c372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 16:32:17.470729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:17.470693 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 16:32:17.470915 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:17.470749 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" podUID="006df6fa-0ef9-4007-869e-0a251d92c372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 16:32:17.579899 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:17.579868 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-67b6f84467-248zx" Oct 30 16:32:17.656456 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:17.656381 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-94sgf"] Oct 30 16:32:17.656874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:17.656842 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" podUID="310ab042-f53a-48f1-8e4a-e8d4e2feb58c" containerName="manager" containerID="cri-o://b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036" gracePeriod=30 Oct 30 16:32:17.913266 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:17.913239 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" Oct 30 16:32:18.064930 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.064837 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-kube-api-access-8tljn\" (UniqueName: \"kubernetes.io/projected/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-kube-api-access-8tljn\") pod \"310ab042-f53a-48f1-8e4a-e8d4e2feb58c\" (UID: \"310ab042-f53a-48f1-8e4a-e8d4e2feb58c\") " Oct 30 16:32:18.065121 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.065030 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-metrics-certs\") pod \"310ab042-f53a-48f1-8e4a-e8d4e2feb58c\" (UID: \"310ab042-f53a-48f1-8e4a-e8d4e2feb58c\") " Oct 30 16:32:18.067185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.067154 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-metrics-certs" pod "310ab042-f53a-48f1-8e4a-e8d4e2feb58c" (UID: "310ab042-f53a-48f1-8e4a-e8d4e2feb58c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:18.067305 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.067231 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-kube-api-access-8tljn" pod "310ab042-f53a-48f1-8e4a-e8d4e2feb58c" (UID: "310ab042-f53a-48f1-8e4a-e8d4e2feb58c"). InnerVolumeSpecName "kube-api-access-8tljn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:32:18.165917 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.165877 5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-metrics-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:32:18.165917 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.165910 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tljn\" (UniqueName: \"kubernetes.io/projected/310ab042-f53a-48f1-8e4a-e8d4e2feb58c-kube-api-access-8tljn\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:32:18.661910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.661813 5838 generic.go:356] "Generic (PLEG): container finished" podID="310ab042-f53a-48f1-8e4a-e8d4e2feb58c" containerID="b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036" exitCode=0 Oct 30 16:32:18.661910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.661881 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" Oct 30 16:32:18.662433 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.661907 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" event={"ID":"310ab042-f53a-48f1-8e4a-e8d4e2feb58c","Type":"ContainerDied","Data":"b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036"} Oct 30 16:32:18.662433 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.662000 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-94sgf" event={"ID":"310ab042-f53a-48f1-8e4a-e8d4e2feb58c","Type":"ContainerDied","Data":"9e8b17b0d7d94a90e59151dc393ec11e8b649a8bea3b4fa5e6935a42c14bd312"} Oct 30 16:32:18.662433 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.662042 5838 scope.go:122] "RemoveContainer" containerID="b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036" Oct 30 16:32:18.672581 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.672565 5838 scope.go:122] "RemoveContainer" containerID="b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036" Oct 30 16:32:18.672886 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:32:18.672848 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036\": container with ID starting with b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036 not found: ID does not exist" containerID="b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036" Oct 30 16:32:18.673090 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.672885 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036"} err="failed to get container status \"b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036\": rpc error: code = NotFound desc = could not find container \"b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036\": container with ID starting with b28a8f1a03bfb5b5aebcb7d879371a084bc5b15d563b4ce67a76f7039996c036 not found: ID does not exist" Oct 30 16:32:18.722062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.722022 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-94sgf"] Oct 30 16:32:18.737043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:18.736739 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-94sgf"] Oct 30 16:32:19.203004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:19.202955 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:19.203483 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:19.203457 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:19.204316 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:19.204300 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" Oct 30 16:32:20.393987 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:20.393935 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310ab042-f53a-48f1-8e4a-e8d4e2feb58c" path="/var/lib/kubelet/pods/310ab042-f53a-48f1-8e4a-e8d4e2feb58c/volumes" Oct 30 16:32:31.797010 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:32:31.796954 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:32:47.676077 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.676039 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd"] Oct 30 16:32:47.677294 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.677260 5838 cpu_manager.go:414] "containerMap: removing container" podUID="310ab042-f53a-48f1-8e4a-e8d4e2feb58c" containerName="manager" Oct 30 16:32:47.677294 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.677288 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310ab042-f53a-48f1-8e4a-e8d4e2feb58c" containerName="manager" Oct 30 16:32:47.677492 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.677366 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="310ab042-f53a-48f1-8e4a-e8d4e2feb58c" containerName="manager" Oct 30 16:32:47.687273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.687236 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.705495 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.705459 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd"] Oct 30 16:32:47.714631 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.714600 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b55521-openshift-default-dockercfg-wtvp5\"" Oct 30 16:32:47.719148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.719114 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-envoy\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.719318 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.719158 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-data\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.719318 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.719235 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-podinfo\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.719318 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.719269 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad79af6-22c9-4997-a3cf-2090794ce3e9-istiod-ca-cert\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.719318 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.719303 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-credential-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.719477 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.719395 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-token\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.719477 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.719447 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pchxl\" (UniqueName: \"kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-kube-api-access-pchxl\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.719551 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.719520 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.719551 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.719538 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-certs\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.820625 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.820584 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.820625 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.820630 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-certs\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.820867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.820688 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-envoy\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.820867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.820707 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-data\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.820867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.820722 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-podinfo\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.820867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.820747 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad79af6-22c9-4997-a3cf-2090794ce3e9-istiod-ca-cert\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.820867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.820767 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-credential-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.820867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.820814 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-token\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.820867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.820860 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pchxl\" (UniqueName: \"kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-kube-api-access-pchxl\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.821615 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.821583 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.821750 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.821640 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-data\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.822239 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.821896 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-credential-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.822239 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.822061 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-certs\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.822239 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.822105 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad79af6-22c9-4997-a3cf-2090794ce3e9-istiod-ca-cert\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.824011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.823844 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-envoy\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.824259 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.824239 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-podinfo\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.858688 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.858645 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-token\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:47.858688 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:47.858672 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pchxl\" (UniqueName: \"kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-kube-api-access-pchxl\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:48.011186 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.011140 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:48.141573 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.141538 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc"] Oct 30 16:32:48.146940 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.146910 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.166388 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.166353 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a7610c-openshift-default-dockercfg-8hbdj\"" Oct 30 16:32:48.173158 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.173072 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc"] Oct 30 16:32:48.225114 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.225073 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340ac92b-1be2-4337-a547-c0c0d239ca96-istiod-ca-cert\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.225296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.225131 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.225296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.225210 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq5j2\" (UniqueName: \"kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-kube-api-access-fq5j2\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.225296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.225247 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-token\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.225296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.225279 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-credential-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.225429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.225337 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-certs\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.225475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.225456 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-data\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.225530 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.225514 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-envoy\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.225573 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.225562 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-podinfo\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.257758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.257720 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw"] Oct 30 16:32:48.262127 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.262030 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.284573 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.284536 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw"] Oct 30 16:32:48.326266 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326220 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-certs\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.326454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326280 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-envoy\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.326454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326354 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-podinfo\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.326454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326391 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340ac92b-1be2-4337-a547-c0c0d239ca96-istiod-ca-cert\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.326454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326426 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-credential-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.326619 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326462 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-token\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.326619 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326493 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.326619 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326587 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq5j2\" (UniqueName: \"kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-kube-api-access-fq5j2\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.326738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326644 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-envoy\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.326738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326681 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-token\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.326738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326707 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-data\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.326872 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326744 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-credential-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.326872 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326769 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-certs\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.326872 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326808 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istiod-ca-cert\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.326872 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326835 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-podinfo\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.327042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326883 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z8mz\" (UniqueName: \"kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-kube-api-access-8z8mz\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.327042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326920 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-data\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.327042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.326965 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.327192 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.327173 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.327226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.327177 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340ac92b-1be2-4337-a547-c0c0d239ca96-istiod-ca-cert\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.327431 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.327412 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-certs\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.327558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.327534 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-data\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.327627 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.327611 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-credential-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.328804 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.328785 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-envoy\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.329069 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.329051 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-podinfo\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.331380 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.331355 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b8ab9f-openshift-default-dockercfg-lh4z4\"" Oct 30 16:32:48.344098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.344059 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd"] Oct 30 16:32:48.348924 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:48.348878 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad79af6_22c9_4997_a3cf_2090794ce3e9.slice/crio-533a0f717db6fe519b3aa46297fdcee38cca70059d3b65f567cc98072054e0f0 WatchSource:0}: Error finding container 533a0f717db6fe519b3aa46297fdcee38cca70059d3b65f567cc98072054e0f0: Status 404 returned error can't find the container with id 533a0f717db6fe519b3aa46297fdcee38cca70059d3b65f567cc98072054e0f0 Oct 30 16:32:48.350411 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.350383 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-token\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.350578 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.350524 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq5j2\" (UniqueName: \"kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-kube-api-access-fq5j2\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.351275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.351245 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:48.351340 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.351312 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:48.351374 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.351343 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:48.428087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.428040 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-credential-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.428378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.428107 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-token\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.428378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.428187 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-envoy\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.428378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.428224 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-data\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.428378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.428266 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istiod-ca-cert\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.428378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.428303 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-podinfo\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.428378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.428367 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z8mz\" (UniqueName: \"kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-kube-api-access-8z8mz\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.428948 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.428417 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.428948 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.428466 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-certs\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.428948 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.428914 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-certs\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.429585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.429563 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istiod-ca-cert\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.430351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.430325 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-credential-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.430605 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.430583 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-data\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.431371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.431331 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.434287 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.434257 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-envoy\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.434506 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.434486 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-podinfo\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.453348 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.453193 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-token\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.455417 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.455365 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z8mz\" (UniqueName: \"kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-kube-api-access-8z8mz\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.461531 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.461494 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:48.581494 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.580613 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:48.817315 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.817227 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc"] Oct 30 16:32:48.820394 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:48.820347 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340ac92b_1be2_4337_a547_c0c0d239ca96.slice/crio-071f0f84be996dfe67b4ec6cd574b5330aff5337a5fd4e6d6b0786b5569f3b52 WatchSource:0}: Error finding container 071f0f84be996dfe67b4ec6cd574b5330aff5337a5fd4e6d6b0786b5569f3b52: Status 404 returned error can't find the container with id 071f0f84be996dfe67b4ec6cd574b5330aff5337a5fd4e6d6b0786b5569f3b52 Oct 30 16:32:48.822810 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.822768 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:48.822893 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.822859 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:48.822939 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.822901 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:48.823916 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.823888 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" event={"ID":"bad79af6-22c9-4997-a3cf-2090794ce3e9","Type":"ContainerStarted","Data":"d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe"} Oct 30 16:32:48.824025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.823961 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" event={"ID":"bad79af6-22c9-4997-a3cf-2090794ce3e9","Type":"ContainerStarted","Data":"533a0f717db6fe519b3aa46297fdcee38cca70059d3b65f567cc98072054e0f0"} Oct 30 16:32:48.873363 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.873256 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" podStartSLOduration=1.8732358740000001 podStartE2EDuration="1.873235874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:48.871305903 +0000 UTC m=+730.619895290" watchObservedRunningTime="2026-10-30 16:32:48.873235874 +0000 UTC m=+730.621825261" Oct 30 16:32:48.896618 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.896577 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw"] Oct 30 16:32:48.910366 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:48.910326 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6f8643_fe14_47eb_81f1_bd89bd12bdbe.slice/crio-c6ac3c2eee2110dd7d8febae50c2acdc200358da24d00d6884453a6284e78c13 WatchSource:0}: Error finding container c6ac3c2eee2110dd7d8febae50c2acdc200358da24d00d6884453a6284e78c13: Status 404 returned error can't find the container with id c6ac3c2eee2110dd7d8febae50c2acdc200358da24d00d6884453a6284e78c13 Oct 30 16:32:48.913464 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.913426 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:48.913583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.913515 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:48.913583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:48.913563 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:32:49.011795 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:49.011674 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:49.022807 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:49.022509 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:49.830786 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:49.830754 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" event={"ID":"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe","Type":"ContainerStarted","Data":"73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81"} Oct 30 16:32:49.831253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:49.830807 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" event={"ID":"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe","Type":"ContainerStarted","Data":"c6ac3c2eee2110dd7d8febae50c2acdc200358da24d00d6884453a6284e78c13"} Oct 30 16:32:49.832475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:49.832448 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" event={"ID":"340ac92b-1be2-4337-a547-c0c0d239ca96","Type":"ContainerStarted","Data":"84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c"} Oct 30 16:32:49.832585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:49.832490 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" event={"ID":"340ac92b-1be2-4337-a547-c0c0d239ca96","Type":"ContainerStarted","Data":"071f0f84be996dfe67b4ec6cd574b5330aff5337a5fd4e6d6b0786b5569f3b52"} Oct 30 16:32:49.832722 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:49.832707 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:49.833774 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:49.833757 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:32:49.874936 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:49.874889 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" podStartSLOduration=1.874875042 podStartE2EDuration="1.874875042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:49.873598393 +0000 UTC m=+731.622187801" watchObservedRunningTime="2026-10-30 16:32:49.874875042 +0000 UTC m=+731.623464429" Oct 30 16:32:49.962577 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:49.962452 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" podStartSLOduration=1.9624331000000002 podStartE2EDuration="1.9624331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:49.961694422 +0000 UTC m=+731.710283834" watchObservedRunningTime="2026-10-30 16:32:49.9624331 +0000 UTC m=+731.711022488" Oct 30 16:32:50.462371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:50.462326 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:50.467671 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:50.467643 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:50.582564 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:50.582526 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:50.588455 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:50.588427 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:50.838516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:50.838481 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:50.839023 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:50.838540 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:50.839260 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:50.839235 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:32:50.839362 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:50.839299 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:32:53.122290 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.122251 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9"] Oct 30 16:32:53.126509 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.126484 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" Oct 30 16:32:53.140899 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.140862 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-ccf5c9\"" Oct 30 16:32:53.152294 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.152267 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-ccf5c9-dockercfg-tvlgl\"" Oct 30 16:32:53.155739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.155702 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9"] Oct 30 16:32:53.198643 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.198587 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde47769-223e-49c6-b4e9-b5c9295d8d85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9\" (UID: \"fde47769-223e-49c6-b4e9-b5c9295d8d85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" Oct 30 16:32:53.198836 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.198716 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmk8f\" (UniqueName: \"kubernetes.io/projected/fde47769-223e-49c6-b4e9-b5c9295d8d85-kube-api-access-hmk8f\") pod \"payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9\" (UID: \"fde47769-223e-49c6-b4e9-b5c9295d8d85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" Oct 30 16:32:53.227268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.210648 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr"] Oct 30 16:32:53.227268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.219930 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" Oct 30 16:32:53.238944 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.238296 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr"] Oct 30 16:32:53.299733 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.299689 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde47769-223e-49c6-b4e9-b5c9295d8d85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9\" (UID: \"fde47769-223e-49c6-b4e9-b5c9295d8d85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" Oct 30 16:32:53.299928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.299759 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmk8f\" (UniqueName: \"kubernetes.io/projected/fde47769-223e-49c6-b4e9-b5c9295d8d85-kube-api-access-hmk8f\") pod \"payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9\" (UID: \"fde47769-223e-49c6-b4e9-b5c9295d8d85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" Oct 30 16:32:53.299928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.299782 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d91ca0-9721-4ff9-8831-18a73e08b207-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr\" (UID: \"a4d91ca0-9721-4ff9-8831-18a73e08b207\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" Oct 30 16:32:53.299928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.299824 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz9wj\" (UniqueName: \"kubernetes.io/projected/a4d91ca0-9721-4ff9-8831-18a73e08b207-kube-api-access-pz9wj\") pod \"payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr\" (UID: \"a4d91ca0-9721-4ff9-8831-18a73e08b207\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" Oct 30 16:32:53.300480 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.300459 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde47769-223e-49c6-b4e9-b5c9295d8d85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9\" (UID: \"fde47769-223e-49c6-b4e9-b5c9295d8d85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" Oct 30 16:32:53.322722 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.322684 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmk8f\" (UniqueName: \"kubernetes.io/projected/fde47769-223e-49c6-b4e9-b5c9295d8d85-kube-api-access-hmk8f\") pod \"payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9\" (UID: \"fde47769-223e-49c6-b4e9-b5c9295d8d85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" Oct 30 16:32:53.400611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.400518 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9wj\" (UniqueName: \"kubernetes.io/projected/a4d91ca0-9721-4ff9-8831-18a73e08b207-kube-api-access-pz9wj\") pod \"payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr\" (UID: \"a4d91ca0-9721-4ff9-8831-18a73e08b207\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" Oct 30 16:32:53.400798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.400673 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d91ca0-9721-4ff9-8831-18a73e08b207-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr\" (UID: \"a4d91ca0-9721-4ff9-8831-18a73e08b207\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" Oct 30 16:32:53.401388 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.401362 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d91ca0-9721-4ff9-8831-18a73e08b207-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr\" (UID: \"a4d91ca0-9721-4ff9-8831-18a73e08b207\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" Oct 30 16:32:53.424081 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.424044 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9wj\" (UniqueName: \"kubernetes.io/projected/a4d91ca0-9721-4ff9-8831-18a73e08b207-kube-api-access-pz9wj\") pod \"payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr\" (UID: \"a4d91ca0-9721-4ff9-8831-18a73e08b207\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" Oct 30 16:32:53.440173 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.440132 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" Oct 30 16:32:53.562001 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.561909 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" Oct 30 16:32:53.742237 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.742189 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9"] Oct 30 16:32:53.746886 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:53.746855 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde47769_223e_49c6_b4e9_b5c9295d8d85.slice/crio-58bfb7657681b4870bbf11729432268f787bef3bfb1acae029656db9163af352 WatchSource:0}: Error finding container 58bfb7657681b4870bbf11729432268f787bef3bfb1acae029656db9163af352: Status 404 returned error can't find the container with id 58bfb7657681b4870bbf11729432268f787bef3bfb1acae029656db9163af352 Oct 30 16:32:53.856605 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.856567 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" event={"ID":"fde47769-223e-49c6-b4e9-b5c9295d8d85","Type":"ContainerStarted","Data":"57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885"} Oct 30 16:32:53.856786 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.856630 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" event={"ID":"fde47769-223e-49c6-b4e9-b5c9295d8d85","Type":"ContainerStarted","Data":"58bfb7657681b4870bbf11729432268f787bef3bfb1acae029656db9163af352"} Oct 30 16:32:53.856786 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.856703 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" Oct 30 16:32:53.864349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.864314 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr"] Oct 30 16:32:53.868564 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:53.868534 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d91ca0_9721_4ff9_8831_18a73e08b207.slice/crio-8d1c4e59b2996c9c7232dcd9b55c86cd3243b560b1e41ef41d57e2e22cc1044b WatchSource:0}: Error finding container 8d1c4e59b2996c9c7232dcd9b55c86cd3243b560b1e41ef41d57e2e22cc1044b: Status 404 returned error can't find the container with id 8d1c4e59b2996c9c7232dcd9b55c86cd3243b560b1e41ef41d57e2e22cc1044b Oct 30 16:32:53.898667 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:53.898611 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" podStartSLOduration=0.898595937 podStartE2EDuration="898.595937ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:53.897000363 +0000 UTC m=+735.645589749" watchObservedRunningTime="2026-10-30 16:32:53.898595937 +0000 UTC m=+735.647185325" Oct 30 16:32:54.654149 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.654111 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p"] Oct 30 16:32:54.658025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.657996 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" Oct 30 16:32:54.676402 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.676352 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b55521\"" Oct 30 16:32:54.678853 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.678823 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p"] Oct 30 16:32:54.712943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.712904 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4"] Oct 30 16:32:54.717421 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.717398 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" Oct 30 16:32:54.721235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.721208 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8pgr\" (UniqueName: \"kubernetes.io/projected/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-kube-api-access-w8pgr\") pod \"payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p\" (UID: \"7203fda6-f9e6-4d6f-96da-0514b72e7ca7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" Oct 30 16:32:54.721371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.721282 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p\" (UID: \"7203fda6-f9e6-4d6f-96da-0514b72e7ca7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" Oct 30 16:32:54.733891 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.733856 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b55521-dockercfg-bcgqt\"" Oct 30 16:32:54.736024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.735998 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4"] Oct 30 16:32:54.822400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.822347 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p\" (UID: \"7203fda6-f9e6-4d6f-96da-0514b72e7ca7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" Oct 30 16:32:54.822554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.822435 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4\" (UID: \"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" Oct 30 16:32:54.822554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.822527 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8pgr\" (UniqueName: \"kubernetes.io/projected/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-kube-api-access-w8pgr\") pod \"payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p\" (UID: \"7203fda6-f9e6-4d6f-96da-0514b72e7ca7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" Oct 30 16:32:54.822657 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.822572 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5hc\" (UniqueName: \"kubernetes.io/projected/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-kube-api-access-tw5hc\") pod \"payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4\" (UID: \"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" Oct 30 16:32:54.826224 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.823704 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p\" (UID: \"7203fda6-f9e6-4d6f-96da-0514b72e7ca7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" Oct 30 16:32:54.847003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.846953 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8pgr\" (UniqueName: \"kubernetes.io/projected/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-kube-api-access-w8pgr\") pod \"payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p\" (UID: \"7203fda6-f9e6-4d6f-96da-0514b72e7ca7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" Oct 30 16:32:54.863228 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.863191 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" event={"ID":"a4d91ca0-9721-4ff9-8831-18a73e08b207","Type":"ContainerStarted","Data":"ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0"} Oct 30 16:32:54.863403 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.863254 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" event={"ID":"a4d91ca0-9721-4ff9-8831-18a73e08b207","Type":"ContainerStarted","Data":"8d1c4e59b2996c9c7232dcd9b55c86cd3243b560b1e41ef41d57e2e22cc1044b"} Oct 30 16:32:54.863497 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.863425 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" Oct 30 16:32:54.903548 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.903471 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" podStartSLOduration=1.903449226 podStartE2EDuration="1.903449226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:54.901604838 +0000 UTC m=+736.650194244" watchObservedRunningTime="2026-10-30 16:32:54.903449226 +0000 UTC m=+736.652038615" Oct 30 16:32:54.924129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.924044 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4\" (UID: \"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" Oct 30 16:32:54.924281 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.924157 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5hc\" (UniqueName: \"kubernetes.io/projected/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-kube-api-access-tw5hc\") pod \"payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4\" (UID: \"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" Oct 30 16:32:54.924908 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.924885 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4\" (UID: \"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" Oct 30 16:32:54.947194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.947160 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5hc\" (UniqueName: \"kubernetes.io/projected/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-kube-api-access-tw5hc\") pod \"payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4\" (UID: \"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" Oct 30 16:32:54.971837 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:54.971785 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" Oct 30 16:32:55.032063 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:55.031745 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" Oct 30 16:32:55.291876 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:55.291844 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p"] Oct 30 16:32:55.295344 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:55.295315 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7203fda6_f9e6_4d6f_96da_0514b72e7ca7.slice/crio-1787f4f0e8e1599ee28fb0c17c8bdb102b06f3c658d6386ddcd8943b7bd621f8 WatchSource:0}: Error finding container 1787f4f0e8e1599ee28fb0c17c8bdb102b06f3c658d6386ddcd8943b7bd621f8: Status 404 returned error can't find the container with id 1787f4f0e8e1599ee28fb0c17c8bdb102b06f3c658d6386ddcd8943b7bd621f8 Oct 30 16:32:55.352871 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:55.352840 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4"] Oct 30 16:32:55.366199 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:55.366164 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0795bf8a_e9c9_440f_97a1_d2fdaa47f97f.slice/crio-3a781507daaba980586ed0ba63b3be9a073ad02f9e8ccf8dcc530ee0f6cbd0be WatchSource:0}: Error finding container 3a781507daaba980586ed0ba63b3be9a073ad02f9e8ccf8dcc530ee0f6cbd0be: Status 404 returned error can't find the container with id 3a781507daaba980586ed0ba63b3be9a073ad02f9e8ccf8dcc530ee0f6cbd0be Oct 30 16:32:55.870186 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:55.870143 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" event={"ID":"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f","Type":"ContainerStarted","Data":"1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163"} Oct 30 16:32:55.870630 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:55.870212 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" event={"ID":"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f","Type":"ContainerStarted","Data":"3a781507daaba980586ed0ba63b3be9a073ad02f9e8ccf8dcc530ee0f6cbd0be"} Oct 30 16:32:55.870630 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:55.870336 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" Oct 30 16:32:55.874564 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:55.874522 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" event={"ID":"7203fda6-f9e6-4d6f-96da-0514b72e7ca7","Type":"ContainerStarted","Data":"e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba"} Oct 30 16:32:55.874750 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:55.874585 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" event={"ID":"7203fda6-f9e6-4d6f-96da-0514b72e7ca7","Type":"ContainerStarted","Data":"1787f4f0e8e1599ee28fb0c17c8bdb102b06f3c658d6386ddcd8943b7bd621f8"} Oct 30 16:32:55.909828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:55.909772 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" podStartSLOduration=1.9097575409999998 podStartE2EDuration="1.909757541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:55.90855335 +0000 UTC m=+737.657142757" watchObservedRunningTime="2026-10-30 16:32:55.909757541 +0000 UTC m=+737.658346925" Oct 30 16:32:55.950226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:55.950172 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" podStartSLOduration=1.950157106 podStartE2EDuration="1.950157106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:55.947791589 +0000 UTC m=+737.696380982" watchObservedRunningTime="2026-10-30 16:32:55.950157106 +0000 UTC m=+737.698746492" Oct 30 16:32:56.749883 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.749841 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p"] Oct 30 16:32:56.756942 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.756919 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" Oct 30 16:32:56.777860 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.777831 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-2e0784\"" Oct 30 16:32:56.778029 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.777904 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-2e0784-dockercfg-znrr6\"" Oct 30 16:32:56.786261 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.786217 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p"] Oct 30 16:32:56.796114 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.796081 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m"] Oct 30 16:32:56.799231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.799213 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" Oct 30 16:32:56.816912 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.816884 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:32:56.819495 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.819464 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m"] Oct 30 16:32:56.845963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.845925 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m\" (UID: \"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" Oct 30 16:32:56.846126 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.845985 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a40be2-e7d1-49f9-a304-2623278732ba-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p\" (UID: \"a1a40be2-e7d1-49f9-a304-2623278732ba\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" Oct 30 16:32:56.846126 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.846063 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87gmp\" (UniqueName: \"kubernetes.io/projected/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-kube-api-access-87gmp\") pod \"payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m\" (UID: \"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" Oct 30 16:32:56.846238 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.846187 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmq6f\" (UniqueName: \"kubernetes.io/projected/a1a40be2-e7d1-49f9-a304-2623278732ba-kube-api-access-pmq6f\") pod \"payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p\" (UID: \"a1a40be2-e7d1-49f9-a304-2623278732ba\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" Oct 30 16:32:56.880316 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.880288 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" Oct 30 16:32:56.947287 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.947248 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m\" (UID: \"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" Oct 30 16:32:56.947472 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.947305 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a40be2-e7d1-49f9-a304-2623278732ba-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p\" (UID: \"a1a40be2-e7d1-49f9-a304-2623278732ba\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" Oct 30 16:32:56.947472 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.947335 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87gmp\" (UniqueName: \"kubernetes.io/projected/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-kube-api-access-87gmp\") pod \"payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m\" (UID: \"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" Oct 30 16:32:56.947585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.947468 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmq6f\" (UniqueName: \"kubernetes.io/projected/a1a40be2-e7d1-49f9-a304-2623278732ba-kube-api-access-pmq6f\") pod \"payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p\" (UID: \"a1a40be2-e7d1-49f9-a304-2623278732ba\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" Oct 30 16:32:56.948305 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.948279 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m\" (UID: \"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" Oct 30 16:32:56.948408 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.948287 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a40be2-e7d1-49f9-a304-2623278732ba-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p\" (UID: \"a1a40be2-e7d1-49f9-a304-2623278732ba\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" Oct 30 16:32:57.000073 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:56.995800 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmq6f\" (UniqueName: \"kubernetes.io/projected/a1a40be2-e7d1-49f9-a304-2623278732ba-kube-api-access-pmq6f\") pod \"payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p\" (UID: \"a1a40be2-e7d1-49f9-a304-2623278732ba\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" Oct 30 16:32:57.013115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.013076 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87gmp\" (UniqueName: \"kubernetes.io/projected/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-kube-api-access-87gmp\") pod \"payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m\" (UID: \"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" Oct 30 16:32:57.071538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.071500 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" Oct 30 16:32:57.111405 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.111369 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" Oct 30 16:32:57.387267 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.387181 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p"] Oct 30 16:32:57.391199 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:57.391126 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a40be2_e7d1_49f9_a304_2623278732ba.slice/crio-d083c6cf5e8aed37d69948015a816a82613cbfff4556bd921af30df0913289ae WatchSource:0}: Error finding container d083c6cf5e8aed37d69948015a816a82613cbfff4556bd921af30df0913289ae: Status 404 returned error can't find the container with id d083c6cf5e8aed37d69948015a816a82613cbfff4556bd921af30df0913289ae Oct 30 16:32:57.429026 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.428993 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m"] Oct 30 16:32:57.433106 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:57.433079 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f49c751_3b87_4bd8_bc8f_cdc4e2dc2a98.slice/crio-bc3bdc49116049c937629187a951392b3c45e4b08dd6e05822c1369617c87f3e WatchSource:0}: Error finding container bc3bdc49116049c937629187a951392b3c45e4b08dd6e05822c1369617c87f3e: Status 404 returned error can't find the container with id bc3bdc49116049c937629187a951392b3c45e4b08dd6e05822c1369617c87f3e Oct 30 16:32:57.640695 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.637663 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj"] Oct 30 16:32:57.641988 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.641950 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz"] Oct 30 16:32:57.642107 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.642070 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" Oct 30 16:32:57.645714 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.645693 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" Oct 30 16:32:57.664066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.664028 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-063dcc\"" Oct 30 16:32:57.666740 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.666669 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj"] Oct 30 16:32:57.669106 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.669082 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz"] Oct 30 16:32:57.773650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.766620 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bc187b-d110-4d74-b2b9-2b96919c1a2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj\" (UID: \"02bc187b-d110-4d74-b2b9-2b96919c1a2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" Oct 30 16:32:57.773650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.766693 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpfxq\" (UniqueName: \"kubernetes.io/projected/02bc187b-d110-4d74-b2b9-2b96919c1a2a-kube-api-access-xpfxq\") pod \"payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj\" (UID: \"02bc187b-d110-4d74-b2b9-2b96919c1a2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" Oct 30 16:32:57.773650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.766859 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdczl\" (UniqueName: \"kubernetes.io/projected/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-kube-api-access-fdczl\") pod \"payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz\" (UID: \"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" Oct 30 16:32:57.773650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.766900 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz\" (UID: \"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" Oct 30 16:32:57.867839 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.867795 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdczl\" (UniqueName: \"kubernetes.io/projected/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-kube-api-access-fdczl\") pod \"payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz\" (UID: \"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" Oct 30 16:32:57.867839 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.867845 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz\" (UID: \"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" Oct 30 16:32:57.868122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.867899 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bc187b-d110-4d74-b2b9-2b96919c1a2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj\" (UID: \"02bc187b-d110-4d74-b2b9-2b96919c1a2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" Oct 30 16:32:57.868122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.867922 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpfxq\" (UniqueName: \"kubernetes.io/projected/02bc187b-d110-4d74-b2b9-2b96919c1a2a-kube-api-access-xpfxq\") pod \"payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj\" (UID: \"02bc187b-d110-4d74-b2b9-2b96919c1a2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" Oct 30 16:32:57.868674 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.868649 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bc187b-d110-4d74-b2b9-2b96919c1a2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj\" (UID: \"02bc187b-d110-4d74-b2b9-2b96919c1a2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" Oct 30 16:32:57.868763 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.868719 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz\" (UID: \"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" Oct 30 16:32:57.894066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.888902 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" event={"ID":"a1a40be2-e7d1-49f9-a304-2623278732ba","Type":"ContainerStarted","Data":"73cccef9c8dece852b7aa5213fd03e58abe3387099d5f53e4cf4867aab0e7d08"} Oct 30 16:32:57.894066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.889012 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" event={"ID":"a1a40be2-e7d1-49f9-a304-2623278732ba","Type":"ContainerStarted","Data":"d083c6cf5e8aed37d69948015a816a82613cbfff4556bd921af30df0913289ae"} Oct 30 16:32:57.894066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.889050 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" Oct 30 16:32:57.894066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.890954 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" event={"ID":"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98","Type":"ContainerStarted","Data":"e77f525c1869f3c4318c540dec0abfc3c9492e9b0506692786cdf96c5caeed5e"} Oct 30 16:32:57.894066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.891029 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" Oct 30 16:32:57.894066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.891059 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" event={"ID":"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98","Type":"ContainerStarted","Data":"bc3bdc49116049c937629187a951392b3c45e4b08dd6e05822c1369617c87f3e"} Oct 30 16:32:57.894066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.891122 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpfxq\" (UniqueName: \"kubernetes.io/projected/02bc187b-d110-4d74-b2b9-2b96919c1a2a-kube-api-access-xpfxq\") pod \"payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj\" (UID: \"02bc187b-d110-4d74-b2b9-2b96919c1a2a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" Oct 30 16:32:57.894066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.891356 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdczl\" (UniqueName: \"kubernetes.io/projected/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-kube-api-access-fdczl\") pod \"payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz\" (UID: \"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" Oct 30 16:32:57.930072 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.930017 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" podStartSLOduration=1.930003948 podStartE2EDuration="1.930003948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:57.929269844 +0000 UTC m=+739.677859244" watchObservedRunningTime="2026-10-30 16:32:57.930003948 +0000 UTC m=+739.678593335" Oct 30 16:32:57.957384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.957352 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" Oct 30 16:32:57.966470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.966445 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" Oct 30 16:32:57.969202 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:57.969150 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" podStartSLOduration=1.969135293 podStartE2EDuration="1.969135293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:57.967772724 +0000 UTC m=+739.716362112" watchObservedRunningTime="2026-10-30 16:32:57.969135293 +0000 UTC m=+739.717724682" Oct 30 16:32:58.271420 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:58.270923 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj"] Oct 30 16:32:58.274102 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:58.274069 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bc187b_d110_4d74_b2b9_2b96919c1a2a.slice/crio-d8c189faee617d31de9566055db10d3a1749dce504b62377e411f8f0919d00c4 WatchSource:0}: Error finding container d8c189faee617d31de9566055db10d3a1749dce504b62377e411f8f0919d00c4: Status 404 returned error can't find the container with id d8c189faee617d31de9566055db10d3a1749dce504b62377e411f8f0919d00c4 Oct 30 16:32:58.287345 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:58.287319 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz"] Oct 30 16:32:58.290107 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:32:58.290085 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4edb3a_808c_4be8_8d9c_c96fb42f5ce3.slice/crio-359d63ac743af59c597c60886d21e510a32617787263071655727086fba5ace7 WatchSource:0}: Error finding container 359d63ac743af59c597c60886d21e510a32617787263071655727086fba5ace7: Status 404 returned error can't find the container with id 359d63ac743af59c597c60886d21e510a32617787263071655727086fba5ace7 Oct 30 16:32:58.899065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:58.899020 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" event={"ID":"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3","Type":"ContainerStarted","Data":"d4590db00d1a7ffe97be06b277da0c9db96b480e55b4a81b844e295353efc916"} Oct 30 16:32:58.899553 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:58.899094 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" event={"ID":"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3","Type":"ContainerStarted","Data":"359d63ac743af59c597c60886d21e510a32617787263071655727086fba5ace7"} Oct 30 16:32:58.899553 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:58.899125 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" Oct 30 16:32:58.900675 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:58.900636 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" event={"ID":"02bc187b-d110-4d74-b2b9-2b96919c1a2a","Type":"ContainerStarted","Data":"b49659f70c245c3be13625c92279396aebe364884c4e4ee8905a335d9839237b"} Oct 30 16:32:58.900792 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:58.900691 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" event={"ID":"02bc187b-d110-4d74-b2b9-2b96919c1a2a","Type":"ContainerStarted","Data":"d8c189faee617d31de9566055db10d3a1749dce504b62377e411f8f0919d00c4"} Oct 30 16:32:58.900861 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:58.900844 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" Oct 30 16:32:58.938407 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:58.938335 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" podStartSLOduration=1.938292674 podStartE2EDuration="1.938292674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:58.937589283 +0000 UTC m=+740.686178670" watchObservedRunningTime="2026-10-30 16:32:58.938292674 +0000 UTC m=+740.686882061" Oct 30 16:32:58.985615 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:32:58.985563 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" podStartSLOduration=1.9855502280000001 podStartE2EDuration="1.985550228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:58.984039288 +0000 UTC m=+740.732628676" watchObservedRunningTime="2026-10-30 16:32:58.985550228 +0000 UTC m=+740.734139614" Oct 30 16:33:00.346577 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.346544 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:33:00.555622 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.555589 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv"] Oct 30 16:33:00.560707 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.560681 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" Oct 30 16:33:00.578375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.578349 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-3e4f91-dockercfg-cwl2f\"" Oct 30 16:33:00.580935 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.580911 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv"] Oct 30 16:33:00.589813 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.589761 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-3e4f91\"" Oct 30 16:33:00.703422 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.703309 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74240496-5b00-44b2-a817-943efddfd6f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv\" (UID: \"74240496-5b00-44b2-a817-943efddfd6f9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" Oct 30 16:33:00.703593 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.703463 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsvxv\" (UniqueName: \"kubernetes.io/projected/74240496-5b00-44b2-a817-943efddfd6f9-kube-api-access-lsvxv\") pod \"payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv\" (UID: \"74240496-5b00-44b2-a817-943efddfd6f9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" Oct 30 16:33:00.804924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.804886 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsvxv\" (UniqueName: \"kubernetes.io/projected/74240496-5b00-44b2-a817-943efddfd6f9-kube-api-access-lsvxv\") pod \"payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv\" (UID: \"74240496-5b00-44b2-a817-943efddfd6f9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" Oct 30 16:33:00.805140 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.805075 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74240496-5b00-44b2-a817-943efddfd6f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv\" (UID: \"74240496-5b00-44b2-a817-943efddfd6f9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" Oct 30 16:33:00.805823 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.805804 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74240496-5b00-44b2-a817-943efddfd6f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv\" (UID: \"74240496-5b00-44b2-a817-943efddfd6f9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" Oct 30 16:33:00.834746 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.834715 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsvxv\" (UniqueName: \"kubernetes.io/projected/74240496-5b00-44b2-a817-943efddfd6f9-kube-api-access-lsvxv\") pod \"payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv\" (UID: \"74240496-5b00-44b2-a817-943efddfd6f9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" Oct 30 16:33:00.873755 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:00.873705 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" Oct 30 16:33:01.244011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:01.241135 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv"] Oct 30 16:33:01.796372 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:33:01.796318 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:33:01.942110 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:01.942065 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" event={"ID":"74240496-5b00-44b2-a817-943efddfd6f9","Type":"ContainerStarted","Data":"eeac892295ea2b9f8345356d80aaf6ed8830710d7b60eed422af8d3ee5e18c61"} Oct 30 16:33:01.942293 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:01.942137 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" event={"ID":"74240496-5b00-44b2-a817-943efddfd6f9","Type":"ContainerStarted","Data":"6eb225190a64a32849ba2fa8c32a3210e5f85777f802cad822aa2a0509ce8b9e"} Oct 30 16:33:01.942293 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:01.942197 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" Oct 30 16:33:01.984641 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:01.984589 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" podStartSLOduration=1.984573376 podStartE2EDuration="1.984573376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:01.984243935 +0000 UTC m=+743.732833343" watchObservedRunningTime="2026-10-30 16:33:01.984573376 +0000 UTC m=+743.733162776" Oct 30 16:33:02.396426 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.396397 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt"] Oct 30 16:33:02.400473 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.400445 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:33:02.437994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.436484 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a7610c\"" Oct 30 16:33:02.437994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.436626 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a7610c-dockercfg-qcnc8\"" Oct 30 16:33:02.445013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.443503 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt"] Oct 30 16:33:02.508008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.505769 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2"] Oct 30 16:33:02.511278 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.511250 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:33:02.547404 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.547313 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2"] Oct 30 16:33:02.559006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.552565 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-826ft\" (UniqueName: \"kubernetes.io/projected/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-kube-api-access-826ft\") pod \"payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt\" (UID: \"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:33:02.559301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.559271 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt\" (UID: \"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:33:02.660650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.660560 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt\" (UID: \"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:33:02.660886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.660668 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bdbw\" (UniqueName: \"kubernetes.io/projected/594c758e-bf83-4687-bb4f-32d89ed6ee53-kube-api-access-2bdbw\") pod \"payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2\" (UID: \"594c758e-bf83-4687-bb4f-32d89ed6ee53\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:33:02.660886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.660713 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-826ft\" (UniqueName: \"kubernetes.io/projected/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-kube-api-access-826ft\") pod \"payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt\" (UID: \"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:33:02.660886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.660821 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594c758e-bf83-4687-bb4f-32d89ed6ee53-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2\" (UID: \"594c758e-bf83-4687-bb4f-32d89ed6ee53\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:33:02.661335 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.661298 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt\" (UID: \"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:33:02.687394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.687359 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-826ft\" (UniqueName: \"kubernetes.io/projected/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-kube-api-access-826ft\") pod \"payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt\" (UID: \"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:33:02.738000 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.737191 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:33:02.761841 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.761799 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594c758e-bf83-4687-bb4f-32d89ed6ee53-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2\" (UID: \"594c758e-bf83-4687-bb4f-32d89ed6ee53\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:33:02.762001 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.761984 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bdbw\" (UniqueName: \"kubernetes.io/projected/594c758e-bf83-4687-bb4f-32d89ed6ee53-kube-api-access-2bdbw\") pod \"payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2\" (UID: \"594c758e-bf83-4687-bb4f-32d89ed6ee53\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:33:02.763141 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.763099 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594c758e-bf83-4687-bb4f-32d89ed6ee53-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2\" (UID: \"594c758e-bf83-4687-bb4f-32d89ed6ee53\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:33:02.787270 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.787238 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bdbw\" (UniqueName: \"kubernetes.io/projected/594c758e-bf83-4687-bb4f-32d89ed6ee53-kube-api-access-2bdbw\") pod \"payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2\" (UID: \"594c758e-bf83-4687-bb4f-32d89ed6ee53\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:33:02.889379 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:02.889338 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:33:03.053032 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:03.050747 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt"] Oct 30 16:33:03.241130 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:03.239203 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2"] Oct 30 16:33:03.244547 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:33:03.244410 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594c758e_bf83_4687_bb4f_32d89ed6ee53.slice/crio-d69a92e15688e70ecbd68bb3096c8b3cb53d069fa21304e0855a7771aa9fb84e WatchSource:0}: Error finding container d69a92e15688e70ecbd68bb3096c8b3cb53d069fa21304e0855a7771aa9fb84e: Status 404 returned error can't find the container with id d69a92e15688e70ecbd68bb3096c8b3cb53d069fa21304e0855a7771aa9fb84e Oct 30 16:33:03.956396 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:03.956364 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" event={"ID":"594c758e-bf83-4687-bb4f-32d89ed6ee53","Type":"ContainerStarted","Data":"2743d3299cd1aa288f8bfb861c48eb1ecc60e23e579de2584e09e7f56da4b839"} Oct 30 16:33:03.956806 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:03.956423 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" event={"ID":"594c758e-bf83-4687-bb4f-32d89ed6ee53","Type":"ContainerStarted","Data":"d69a92e15688e70ecbd68bb3096c8b3cb53d069fa21304e0855a7771aa9fb84e"} Oct 30 16:33:03.956806 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:03.956452 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:33:03.958477 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:03.958444 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" event={"ID":"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a","Type":"ContainerStarted","Data":"5489effa2ff750e1c47ca8978d4024d02c4aa6add21b6c6801aae975c404c8ba"} Oct 30 16:33:03.958610 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:03.958494 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" event={"ID":"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a","Type":"ContainerStarted","Data":"745b67335c8a880f189cd92d53bc4a876a93fa38a6feab5af82eb6e4ae4e8abf"} Oct 30 16:33:03.958610 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:03.958572 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:33:04.006086 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:04.006027 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" podStartSLOduration=2.006007519 podStartE2EDuration="2.006007519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:04.003388123 +0000 UTC m=+745.751977510" watchObservedRunningTime="2026-10-30 16:33:04.006007519 +0000 UTC m=+745.754596907" Oct 30 16:33:04.072735 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:04.072634 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" podStartSLOduration=2.072615768 podStartE2EDuration="2.072615768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:04.066652877 +0000 UTC m=+745.815242263" watchObservedRunningTime="2026-10-30 16:33:04.072615768 +0000 UTC m=+745.821205157" Oct 30 16:33:04.866168 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:04.866135 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" Oct 30 16:33:05.877946 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:05.877915 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" Oct 30 16:33:06.883873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:06.883777 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" Oct 30 16:33:07.894736 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:07.894702 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" Oct 30 16:33:08.904146 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:08.904119 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" Oct 30 16:33:08.904572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:08.904191 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" Oct 30 16:33:09.261887 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:09.261841 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:33:09.909801 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:09.909776 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" Oct 30 16:33:09.910174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:09.909905 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" Oct 30 16:33:10.000313 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:10.000283 5838 generic.go:356] "Generic (PLEG): container finished" podID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerID="d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe" exitCode=0 Oct 30 16:33:10.000505 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:10.000354 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" event={"ID":"bad79af6-22c9-4997-a3cf-2090794ce3e9","Type":"ContainerDied","Data":"d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe"} Oct 30 16:33:10.000810 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:10.000769 5838 scope.go:122] "RemoveContainer" containerID="d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe" Oct 30 16:33:10.002181 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:10.002153 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:33:10.002265 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:10.002232 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:33:10.002312 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:10.002263 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:33:11.008186 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:11.008142 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" event={"ID":"bad79af6-22c9-4997-a3cf-2090794ce3e9","Type":"ContainerStarted","Data":"cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6"} Oct 30 16:33:11.011703 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:11.011684 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:33:11.011890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:11.011872 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 16:33:11.011930 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:11.011913 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 16:33:12.011754 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:12.011718 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 16:33:12.012200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:12.011791 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 16:33:12.952348 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:12.952319 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" Oct 30 16:33:13.012103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:13.012066 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 16:33:13.012589 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:13.012132 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 16:33:14.012559 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:14.012524 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 16:33:14.012952 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:14.012578 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 16:33:14.974156 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:14.974103 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:33:14.974379 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:14.974261 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:33:15.012391 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.012350 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 16:33:15.012565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.012421 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 16:33:15.245625 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.245589 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx"] Oct 30 16:33:15.250254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.250221 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.264005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.263962 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx"] Oct 30 16:33:15.266548 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.266522 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-a7610c-serving-cert\"" Oct 30 16:33:15.266677 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.266651 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:33:15.267004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.266963 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:33:15.267101 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.267068 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dqwfr\"" Oct 30 16:33:15.267178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.267144 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 16:33:15.313465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.313416 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33384be7-b235-48c3-9cda-dad084502291-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.313465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.313456 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-metrics-certs\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.313734 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.313676 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.313796 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.313774 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bplr\" (UniqueName: \"kubernetes.io/projected/33384be7-b235-48c3-9cda-dad084502291-kube-api-access-8bplr\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.418368 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.417633 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.418368 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.417698 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bplr\" (UniqueName: \"kubernetes.io/projected/33384be7-b235-48c3-9cda-dad084502291-kube-api-access-8bplr\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.418368 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.417780 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33384be7-b235-48c3-9cda-dad084502291-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.418368 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.417809 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-metrics-certs\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.426614 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:33:15.426580 5838 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-serving-cert: secret "maas-api-e2e-worker-w0-a7610c-serving-cert" not found Oct 30 16:33:15.427633 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:33:15.427614 5838 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-maas-api-tls podName:33384be7-b235-48c3-9cda-dad084502291 nodeName:}" failed. No retries permitted until 2026-10-30 16:33:15.927592159 +0000 UTC m=+757.676181541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-maas-api-tls") pod "maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" (UID: "33384be7-b235-48c3-9cda-dad084502291") : secret "maas-api-e2e-worker-w0-a7610c-serving-cert" not found Oct 30 16:33:15.428072 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.427513 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33384be7-b235-48c3-9cda-dad084502291-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.449669 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.449594 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-metrics-certs\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.455884 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.455775 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bplr\" (UniqueName: \"kubernetes.io/projected/33384be7-b235-48c3-9cda-dad084502291-kube-api-access-8bplr\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.944998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.944265 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:15.967398 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:15.967352 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:16.012138 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:16.012104 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 16:33:16.012310 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:16.012170 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 16:33:16.168026 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:16.167964 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:16.539635 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:16.539530 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx"] Oct 30 16:33:16.543502 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:33:16.543477 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33384be7_b235_48c3_9cda_dad084502291.slice/crio-6d0947ee08d1f405b0d0656a0b95e2a6de509d16e9755be1215bafff1271f7e9 WatchSource:0}: Error finding container 6d0947ee08d1f405b0d0656a0b95e2a6de509d16e9755be1215bafff1271f7e9: Status 404 returned error can't find the container with id 6d0947ee08d1f405b0d0656a0b95e2a6de509d16e9755be1215bafff1271f7e9 Oct 30 16:33:16.545317 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:16.545298 5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:33:17.030000 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:17.029313 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:33:17.031073 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:17.030897 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:33:17.039291 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:17.039263 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:33:17.078247 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:17.078209 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" event={"ID":"33384be7-b235-48c3-9cda-dad084502291","Type":"ContainerStarted","Data":"6d0947ee08d1f405b0d0656a0b95e2a6de509d16e9755be1215bafff1271f7e9"} Oct 30 16:33:20.098418 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:20.098320 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" event={"ID":"33384be7-b235-48c3-9cda-dad084502291","Type":"ContainerStarted","Data":"513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c"} Oct 30 16:33:20.098418 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:20.098418 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:20.142110 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:20.142056 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" podStartSLOduration=1.918280358 podStartE2EDuration="5.142042286s" totalImagesPullingTime="3.223761928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:20.139868975 +0000 UTC m=+761.888458385" watchObservedRunningTime="2026-10-30 16:33:20.142042286 +0000 UTC m=+761.890631672" Oct 30 16:33:21.070465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:21.070427 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:33:26.109948 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:26.109918 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:33:26.169323 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:26.169291 5838 generic.go:356] "Generic (PLEG): container finished" podID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerID="73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81" exitCode=0 Oct 30 16:33:26.169532 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:26.169340 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" event={"ID":"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe","Type":"ContainerDied","Data":"73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81"} Oct 30 16:33:26.169789 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:26.169762 5838 scope.go:122] "RemoveContainer" containerID="73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81" Oct 30 16:33:26.171512 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:26.171475 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:33:26.171626 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:26.171568 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:33:26.171626 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:26.171621 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:33:27.188691 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:27.188518 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" event={"ID":"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe","Type":"ContainerStarted","Data":"c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a"} Oct 30 16:33:27.582951 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:27.582919 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:33:27.583179 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:27.582962 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:33:27.583179 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:27.583029 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 16:33:28.583250 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:28.583214 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:33:28.583706 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:28.583274 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 16:33:29.582304 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:29.582259 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:33:29.582504 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:29.582329 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 16:33:30.582302 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:30.582258 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:33:30.582793 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:30.582341 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 16:33:31.582829 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:31.582787 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:33:31.583270 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:31.582865 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 16:33:31.797185 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:33:31.797139 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:33:32.168325 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:32.168226 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:33:32.239586 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:32.239553 5838 generic.go:356] "Generic (PLEG): container finished" podID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerID="84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c" exitCode=0 Oct 30 16:33:32.239828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:32.239638 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" event={"ID":"340ac92b-1be2-4337-a547-c0c0d239ca96","Type":"ContainerDied","Data":"84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c"} Oct 30 16:33:32.240225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:32.240205 5838 scope.go:122] "RemoveContainer" containerID="84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c" Oct 30 16:33:32.241719 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:32.241691 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:33:32.241822 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:32.241764 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:33:32.241822 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:32.241802 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:33:32.917005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:32.914274 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:33:32.917005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:32.915301 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:33:32.940040 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:32.933290 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:33:33.248896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:33.248861 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" event={"ID":"340ac92b-1be2-4337-a547-c0c0d239ca96","Type":"ContainerStarted","Data":"7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9"} Oct 30 16:33:33.462310 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:33.462268 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:33:33.463556 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:33.463526 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 16:33:33.463711 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:33.463592 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 16:33:34.462775 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.462743 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 16:33:34.463208 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.462814 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 16:33:34.727739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.721953 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m"] Oct 30 16:33:34.732203 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.732174 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.751113 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.751078 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m"] Oct 30 16:33:34.758963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.758933 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ccf5c9\"/\"default-dockercfg-8lvcr\"" Oct 30 16:33:34.759142 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.759101 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ccf5c9\"/\"kube-root-ca.crt\"" Oct 30 16:33:34.759227 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.759177 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ccf5c9\"/\"openshift-service-ca.crt\"" Oct 30 16:33:34.762928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.762908 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ccf5c9\"/\"premium-w3-ccf5c9-kserve-self-signed-certs\"" Oct 30 16:33:34.855524 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.855487 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-model-cache\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.855710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.855550 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w5lw\" (UniqueName: \"kubernetes.io/projected/855723ad-f4a4-4857-bd8e-0174f9f7d00e-kube-api-access-6w5lw\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.855710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.855644 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-home\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.855710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.855667 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tmp-dir\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.855710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.855697 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-dshm\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.855882 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.855712 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tls-certs\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.958995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.958631 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-model-cache\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.958995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.958724 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w5lw\" (UniqueName: \"kubernetes.io/projected/855723ad-f4a4-4857-bd8e-0174f9f7d00e-kube-api-access-6w5lw\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.958995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.958770 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-home\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.958995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.958804 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tmp-dir\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.958995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.958852 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-dshm\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.958995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.958875 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tls-certs\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.959838 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.959798 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-home\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.960488 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.960460 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tmp-dir\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.961920 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.961863 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-model-cache\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.979884 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.979268 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-dshm\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.983079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.983051 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tls-certs\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:34.983858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:34.983831 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w5lw\" (UniqueName: \"kubernetes.io/projected/855723ad-f4a4-4857-bd8e-0174f9f7d00e-kube-api-access-6w5lw\") pod \"premium-w3-ccf5c9-kserve-68949c48f4-q544m\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:35.062158 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:35.062123 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:35.377075 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:35.377037 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m"] Oct 30 16:33:35.380218 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:33:35.380185 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod855723ad_f4a4_4857_bd8e_0174f9f7d00e.slice/crio-9f07733dad65d826550f95696b1b3790b05b6e853944f0327c4d891327528c57 WatchSource:0}: Error finding container 9f07733dad65d826550f95696b1b3790b05b6e853944f0327c4d891327528c57: Status 404 returned error can't find the container with id 9f07733dad65d826550f95696b1b3790b05b6e853944f0327c4d891327528c57 Oct 30 16:33:35.462025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:35.461999 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 16:33:35.462205 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:35.462056 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 16:33:35.839117 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:35.839074 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:33:36.272526 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:36.272480 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" event={"ID":"855723ad-f4a4-4857-bd8e-0174f9f7d00e","Type":"ContainerStarted","Data":"c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96"} Oct 30 16:33:36.272705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:36.272534 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" event={"ID":"855723ad-f4a4-4857-bd8e-0174f9f7d00e","Type":"ContainerStarted","Data":"9f07733dad65d826550f95696b1b3790b05b6e853944f0327c4d891327528c57"} Oct 30 16:33:36.334997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:36.313949 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podStartSLOduration=2.313929176 podStartE2EDuration="2.313929176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:36.313270359 +0000 UTC m=+778.061859751" watchObservedRunningTime="2026-10-30 16:33:36.313929176 +0000 UTC m=+778.062518564" Oct 30 16:33:36.464210 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:36.464173 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 16:33:36.464409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:36.464257 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 16:33:37.062284 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:37.062251 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:37.462751 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:37.462661 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 16:33:37.462751 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:37.462720 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 16:33:38.714801 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:38.714741 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:33:38.716612 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:38.716589 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:33:44.034294 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.034257 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r"] Oct 30 16:33:44.038865 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.038842 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.058545 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.058516 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r"] Oct 30 16:33:44.066961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.066882 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b8ab9f\"/\"kube-root-ca.crt\"" Oct 30 16:33:44.066961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.066909 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b8ab9f\"/\"openshift-service-ca.crt\"" Oct 30 16:33:44.067186 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.067020 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b8ab9f\"/\"default-dockercfg-t9ckg\"" Oct 30 16:33:44.067186 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.067046 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b8ab9f\"/\"simulator-w4-b8ab9f-kserve-self-signed-certs\"" Oct 30 16:33:44.183536 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.183489 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-dshm\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.183734 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.183570 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tls-certs\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.183734 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.183626 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-home\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.183734 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.183685 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkq5p\" (UniqueName: \"kubernetes.io/projected/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-kube-api-access-qkq5p\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.183931 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.183804 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-model-cache\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.183931 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.183868 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tmp-dir\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.290068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.289760 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tmp-dir\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.290068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.289842 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-dshm\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.290068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.289906 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tls-certs\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.290068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.289991 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-home\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.290068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.290021 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkq5p\" (UniqueName: \"kubernetes.io/projected/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-kube-api-access-qkq5p\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.290466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.290109 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-model-cache\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.293993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.290638 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-model-cache\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.293993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.290913 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tmp-dir\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.293993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.291613 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-home\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.315495 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.314999 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-dshm\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.326420 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.326378 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tls-certs\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.331601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.331574 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkq5p\" (UniqueName: \"kubernetes.io/projected/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-kube-api-access-qkq5p\") pod \"simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.353283 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.353254 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:44.664636 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:44.664548 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r"] Oct 30 16:33:44.667608 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:33:44.667585 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf680ef8d_a3ae_493c_b45f_1de23cd0b63a.slice/crio-f9603903ade4bb3ccd857cb665aa0a5b5bcd2809f4b0c7105a7606ef4ee8d9d7 WatchSource:0}: Error finding container f9603903ade4bb3ccd857cb665aa0a5b5bcd2809f4b0c7105a7606ef4ee8d9d7: Status 404 returned error can't find the container with id f9603903ade4bb3ccd857cb665aa0a5b5bcd2809f4b0c7105a7606ef4ee8d9d7 Oct 30 16:33:45.024543 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:45.024511 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:33:45.062660 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:45.062631 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:45.074639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:45.074608 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:45.338257 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:45.338165 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" event={"ID":"f680ef8d-a3ae-493c-b45f-1de23cd0b63a","Type":"ContainerStarted","Data":"bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea"} Oct 30 16:33:45.338257 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:45.338218 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" event={"ID":"f680ef8d-a3ae-493c-b45f-1de23cd0b63a","Type":"ContainerStarted","Data":"f9603903ade4bb3ccd857cb665aa0a5b5bcd2809f4b0c7105a7606ef4ee8d9d7"} Oct 30 16:33:45.349484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:45.349459 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:33:45.353386 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:45.353366 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:45.378999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:45.378926 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podStartSLOduration=1.378914099 podStartE2EDuration="1.378914099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:45.378481065 +0000 UTC m=+787.127070452" watchObservedRunningTime="2026-10-30 16:33:45.378914099 +0000 UTC m=+787.127503485" Oct 30 16:33:52.614058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.614020 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8"] Oct 30 16:33:52.617918 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.617893 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.643597 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.643546 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b55521\"/\"default-dockercfg-69c6c\"" Oct 30 16:33:52.643799 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.643655 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b55521\"/\"premium-w1-b55521-kserve-self-signed-certs\"" Oct 30 16:33:52.646495 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.646471 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8"] Oct 30 16:33:52.674317 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.674285 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e579322a-2a82-46bd-8959-e75ee51bb57d-tls-certs\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.674469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.674327 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-home\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.674469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.674353 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-dshm\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.674469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.674402 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-tmp-dir\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.674638 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.674601 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-model-cache\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.674711 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.674694 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8tl\" (UniqueName: \"kubernetes.io/projected/e579322a-2a82-46bd-8959-e75ee51bb57d-kube-api-access-kn8tl\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.694934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.694904 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b55521\"/\"kube-root-ca.crt\"" Oct 30 16:33:52.695108 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.694947 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b55521\"/\"openshift-service-ca.crt\"" Oct 30 16:33:52.775705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.775662 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8tl\" (UniqueName: \"kubernetes.io/projected/e579322a-2a82-46bd-8959-e75ee51bb57d-kube-api-access-kn8tl\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.775901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.775730 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e579322a-2a82-46bd-8959-e75ee51bb57d-tls-certs\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.775901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.775769 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-home\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.775901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.775800 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-dshm\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.775901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.775817 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-tmp-dir\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.775901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.775877 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-model-cache\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.776300 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.776272 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-home\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.776371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.776283 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-model-cache\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.776371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.776336 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-tmp-dir\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.778079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.778060 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-dshm\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.778335 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.778318 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e579322a-2a82-46bd-8959-e75ee51bb57d-tls-certs\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.823995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.816823 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8tl\" (UniqueName: \"kubernetes.io/projected/e579322a-2a82-46bd-8959-e75ee51bb57d-kube-api-access-kn8tl\") pod \"premium-w1-b55521-kserve-6f84795d55-nkkq8\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:52.932578 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:52.932474 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:53.236284 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:53.236256 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8"] Oct 30 16:33:53.239603 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:33:53.239577 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode579322a_2a82_46bd_8959_e75ee51bb57d.slice/crio-6dbaaedba3fe4ad3597420fba1a8e4aa8aac9edc3e5f297c496640845869fea3 WatchSource:0}: Error finding container 6dbaaedba3fe4ad3597420fba1a8e4aa8aac9edc3e5f297c496640845869fea3: Status 404 returned error can't find the container with id 6dbaaedba3fe4ad3597420fba1a8e4aa8aac9edc3e5f297c496640845869fea3 Oct 30 16:33:53.394296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:53.394255 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" event={"ID":"e579322a-2a82-46bd-8959-e75ee51bb57d","Type":"ContainerStarted","Data":"232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7"} Oct 30 16:33:53.394296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:53.394297 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" event={"ID":"e579322a-2a82-46bd-8959-e75ee51bb57d","Type":"ContainerStarted","Data":"6dbaaedba3fe4ad3597420fba1a8e4aa8aac9edc3e5f297c496640845869fea3"} Oct 30 16:33:53.435367 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:53.435315 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podStartSLOduration=1.435301674 podStartE2EDuration="1.435301674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:53.43460335 +0000 UTC m=+795.183192742" watchObservedRunningTime="2026-10-30 16:33:53.435301674 +0000 UTC m=+795.183891061" Oct 30 16:33:53.932777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:53.932741 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:33:54.354008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:54.353942 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:54.366127 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:54.366103 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:54.411190 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:54.411158 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:33:57.331723 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:33:57.331689 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:34:01.796819 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:34:01.796777 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:34:02.932828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:02.932784 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:34:02.945103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:02.945080 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:34:03.213624 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.213539 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7"] Oct 30 16:34:03.217961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.217936 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.242842 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.242812 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ccf5c9-openshift-default-dockercfg-h2z4k\"" Oct 30 16:34:03.245068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.245039 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7"] Oct 30 16:34:03.294265 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.294229 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-envoy\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.294442 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.294276 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-podinfo\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.294442 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.294299 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.294442 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.294369 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-credential-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.294621 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.294529 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-token\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.294621 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.294562 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istiod-ca-cert\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.294727 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.294629 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-certs\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.294785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.294750 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-data\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.294834 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.294782 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxs2x\" (UniqueName: \"kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-kube-api-access-lxs2x\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.399633 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.399579 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-certs\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.399836 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.399701 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-data\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.399836 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.399733 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxs2x\" (UniqueName: \"kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-kube-api-access-lxs2x\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.399836 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.399800 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-envoy\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.400057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.399835 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-podinfo\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.400057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.399870 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.400057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.399898 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-credential-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.400057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.399964 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-token\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.400057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.400005 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istiod-ca-cert\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.400304 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.400148 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-certs\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.400409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.400387 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-credential-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.400711 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.400688 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.401021 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.400962 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-data\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.401170 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.401150 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istiod-ca-cert\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.402775 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.402744 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-envoy\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.403477 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.403457 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-podinfo\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.423301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.423276 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-token\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.423510 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.423490 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxs2x\" (UniqueName: \"kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-kube-api-access-lxs2x\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.471786 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.471701 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:34:03.549678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.549645 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:03.870271 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.870211 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7"] Oct 30 16:34:03.873155 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:34:03.873129 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39dd14bb_4cf4_4016_b2b1_59519d96fda7.slice/crio-3742a8dae575400aa7984a643edbd9e1aace7b03851cee349eb2ad48bb134438 WatchSource:0}: Error finding container 3742a8dae575400aa7984a643edbd9e1aace7b03851cee349eb2ad48bb134438: Status 404 returned error can't find the container with id 3742a8dae575400aa7984a643edbd9e1aace7b03851cee349eb2ad48bb134438 Oct 30 16:34:03.875592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.875556 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:03.875693 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.875621 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:03.875693 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:03.875649 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:04.470324 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:04.470289 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" event={"ID":"39dd14bb-4cf4-4016-b2b1-59519d96fda7","Type":"ContainerStarted","Data":"434f9907ef5aedf1f52567f9d1e0335a29ae3bc6582f401c39a98e4f4c0afcc2"} Oct 30 16:34:04.470839 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:04.470350 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" event={"ID":"39dd14bb-4cf4-4016-b2b1-59519d96fda7","Type":"ContainerStarted","Data":"3742a8dae575400aa7984a643edbd9e1aace7b03851cee349eb2ad48bb134438"} Oct 30 16:34:04.541554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:04.541490 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" podStartSLOduration=1.541472258 podStartE2EDuration="1.541472258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:04.525905968 +0000 UTC m=+806.274495385" watchObservedRunningTime="2026-10-30 16:34:04.541472258 +0000 UTC m=+806.290061651" Oct 30 16:34:04.550699 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:04.550674 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:04.552552 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:04.552520 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 16:34:04.552664 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:04.552574 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" podUID="39dd14bb-4cf4-4016-b2b1-59519d96fda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 16:34:05.550269 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:05.550228 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 16:34:05.550749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:05.550292 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" podUID="39dd14bb-4cf4-4016-b2b1-59519d96fda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 16:34:06.550823 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:06.550778 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 16:34:06.551362 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:06.550853 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" podUID="39dd14bb-4cf4-4016-b2b1-59519d96fda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 16:34:07.550431 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:07.550397 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 16:34:07.550615 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:07.550455 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" podUID="39dd14bb-4cf4-4016-b2b1-59519d96fda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 16:34:08.559035 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:08.553380 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 16:34:08.559035 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:08.553465 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" podUID="39dd14bb-4cf4-4016-b2b1-59519d96fda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 16:34:09.550325 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:09.550288 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 16:34:09.550515 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:09.550356 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" podUID="39dd14bb-4cf4-4016-b2b1-59519d96fda7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 16:34:11.364915 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:11.364876 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:11.365674 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:11.365649 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:11.368107 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:11.368080 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:34:15.847518 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:15.847486 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:34:16.601343 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.601303 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57"] Oct 30 16:34:16.607335 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.607303 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.626280 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.626244 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57"] Oct 30 16:34:16.633743 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.633715 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b8ab9f\"/\"premium-w4-b8ab9f-kserve-self-signed-certs\"" Oct 30 16:34:16.691671 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.691629 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fwxq\" (UniqueName: \"kubernetes.io/projected/887e70df-0f8a-4eb0-892f-f3ad6747a911-kube-api-access-2fwxq\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.691873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.691752 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-tmp-dir\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.691873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.691782 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887e70df-0f8a-4eb0-892f-f3ad6747a911-tls-certs\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.692018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.691985 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-dshm\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.692059 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.692035 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-home\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.692092 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.692078 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-model-cache\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.796027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.794725 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-tmp-dir\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.796027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.794773 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887e70df-0f8a-4eb0-892f-f3ad6747a911-tls-certs\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.796027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.794874 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-dshm\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.796027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.794899 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-home\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.796027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.794940 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-model-cache\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.796027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.795026 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fwxq\" (UniqueName: \"kubernetes.io/projected/887e70df-0f8a-4eb0-892f-f3ad6747a911-kube-api-access-2fwxq\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.796027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.795822 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-tmp-dir\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.805237 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.805197 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-home\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.806244 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.806111 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-model-cache\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.814770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.814734 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-dshm\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.838796 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.838710 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887e70df-0f8a-4eb0-892f-f3ad6747a911-tls-certs\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.838796 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.838729 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fwxq\" (UniqueName: \"kubernetes.io/projected/887e70df-0f8a-4eb0-892f-f3ad6747a911-kube-api-access-2fwxq\") pod \"premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:16.920381 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:16.920279 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:17.244331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:17.244287 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57"] Oct 30 16:34:17.247980 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:34:17.247936 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod887e70df_0f8a_4eb0_892f_f3ad6747a911.slice/crio-cc5c62c65642eebbccb5def8f19359ae7e1eb90cee2752e94eda896cbc173fb6 WatchSource:0}: Error finding container cc5c62c65642eebbccb5def8f19359ae7e1eb90cee2752e94eda896cbc173fb6: Status 404 returned error can't find the container with id cc5c62c65642eebbccb5def8f19359ae7e1eb90cee2752e94eda896cbc173fb6 Oct 30 16:34:17.597078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:17.596954 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" event={"ID":"887e70df-0f8a-4eb0-892f-f3ad6747a911","Type":"ContainerStarted","Data":"ce8a20514b06af450e34cd21b2f64d999dee33d015a385ed432060ae7c77eba7"} Oct 30 16:34:17.597078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:17.597010 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" event={"ID":"887e70df-0f8a-4eb0-892f-f3ad6747a911","Type":"ContainerStarted","Data":"cc5c62c65642eebbccb5def8f19359ae7e1eb90cee2752e94eda896cbc173fb6"} Oct 30 16:34:17.637185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:17.637125 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podStartSLOduration=1.637111327 podStartE2EDuration="1.637111327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:17.636814985 +0000 UTC m=+819.385404372" watchObservedRunningTime="2026-10-30 16:34:17.637111327 +0000 UTC m=+819.385700714" Oct 30 16:34:17.920946 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:17.920855 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:23.165805 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.165704 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9"] Oct 30 16:34:23.169702 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.169679 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:23.186514 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.186486 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:34:23.197863 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.197835 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:34:23.200752 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.200729 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9"] Oct 30 16:34:23.267478 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.267440 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbf544c0-brqs9\" (UID: \"bf277455-983a-424b-bbdb-a33f3757ca8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:23.267681 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.267490 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbf544c0-brqs9\" (UID: \"bf277455-983a-424b-bbdb-a33f3757ca8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:23.267681 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.267652 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdck5\" (UniqueName: \"kubernetes.io/projected/bf277455-983a-424b-bbdb-a33f3757ca8d-kube-api-access-fdck5\") pod \"maas-api-revoke-sub-cbf544c0-brqs9\" (UID: \"bf277455-983a-424b-bbdb-a33f3757ca8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:23.369369 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.369334 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbf544c0-brqs9\" (UID: \"bf277455-983a-424b-bbdb-a33f3757ca8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:23.369565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.369383 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbf544c0-brqs9\" (UID: \"bf277455-983a-424b-bbdb-a33f3757ca8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:23.369565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.369449 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdck5\" (UniqueName: \"kubernetes.io/projected/bf277455-983a-424b-bbdb-a33f3757ca8d-kube-api-access-fdck5\") pod \"maas-api-revoke-sub-cbf544c0-brqs9\" (UID: \"bf277455-983a-424b-bbdb-a33f3757ca8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:23.370169 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.370147 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbf544c0-brqs9\" (UID: \"bf277455-983a-424b-bbdb-a33f3757ca8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:23.371650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.371632 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbf544c0-brqs9\" (UID: \"bf277455-983a-424b-bbdb-a33f3757ca8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:23.392390 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.392366 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdck5\" (UniqueName: \"kubernetes.io/projected/bf277455-983a-424b-bbdb-a33f3757ca8d-kube-api-access-fdck5\") pod \"maas-api-revoke-sub-cbf544c0-brqs9\" (UID: \"bf277455-983a-424b-bbdb-a33f3757ca8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:23.394937 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.394910 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn"] Oct 30 16:34:23.398929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.398911 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:23.417962 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.417882 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn"] Oct 30 16:34:23.484177 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.484142 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:23.571261 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.571227 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d156127-l7wrn\" (UID: \"3ce677bf-1156-4ede-ab6a-83b6a9268881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:23.571261 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.571271 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d156127-l7wrn\" (UID: \"3ce677bf-1156-4ede-ab6a-83b6a9268881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:23.571465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.571294 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvh27\" (UniqueName: \"kubernetes.io/projected/3ce677bf-1156-4ede-ab6a-83b6a9268881-kube-api-access-bvh27\") pod \"maas-api-revoke-sub-3d156127-l7wrn\" (UID: \"3ce677bf-1156-4ede-ab6a-83b6a9268881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:23.672231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.672143 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d156127-l7wrn\" (UID: \"3ce677bf-1156-4ede-ab6a-83b6a9268881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:23.672231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.672184 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d156127-l7wrn\" (UID: \"3ce677bf-1156-4ede-ab6a-83b6a9268881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:23.672231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.672206 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvh27\" (UniqueName: \"kubernetes.io/projected/3ce677bf-1156-4ede-ab6a-83b6a9268881-kube-api-access-bvh27\") pod \"maas-api-revoke-sub-3d156127-l7wrn\" (UID: \"3ce677bf-1156-4ede-ab6a-83b6a9268881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:23.672811 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.672789 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d156127-l7wrn\" (UID: \"3ce677bf-1156-4ede-ab6a-83b6a9268881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:23.674567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.674545 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d156127-l7wrn\" (UID: \"3ce677bf-1156-4ede-ab6a-83b6a9268881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:23.694827 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.694796 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvh27\" (UniqueName: \"kubernetes.io/projected/3ce677bf-1156-4ede-ab6a-83b6a9268881-kube-api-access-bvh27\") pod \"maas-api-revoke-sub-3d156127-l7wrn\" (UID: \"3ce677bf-1156-4ede-ab6a-83b6a9268881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:23.716895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.716867 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:23.792178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:23.791885 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9"] Oct 30 16:34:23.794679 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:34:23.794644 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf277455_983a_424b_bbdb_a33f3757ca8d.slice/crio-d39ec6ad0be580b5ef77998958869197289bf48311b7452115fcd660257f1238 WatchSource:0}: Error finding container d39ec6ad0be580b5ef77998958869197289bf48311b7452115fcd660257f1238: Status 404 returned error can't find the container with id d39ec6ad0be580b5ef77998958869197289bf48311b7452115fcd660257f1238 Oct 30 16:34:24.021827 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:24.021794 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn"] Oct 30 16:34:24.024937 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:34:24.024907 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce677bf_1156_4ede_ab6a_83b6a9268881.slice/crio-8477ca0eb9c10f485e2da3904594ad2b7f5085231cbf564783e2bec3960acad7 WatchSource:0}: Error finding container 8477ca0eb9c10f485e2da3904594ad2b7f5085231cbf564783e2bec3960acad7: Status 404 returned error can't find the container with id 8477ca0eb9c10f485e2da3904594ad2b7f5085231cbf564783e2bec3960acad7 Oct 30 16:34:24.658841 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:24.658802 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" event={"ID":"bf277455-983a-424b-bbdb-a33f3757ca8d","Type":"ContainerStarted","Data":"d39ec6ad0be580b5ef77998958869197289bf48311b7452115fcd660257f1238"} Oct 30 16:34:24.660036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:24.660010 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" event={"ID":"3ce677bf-1156-4ede-ab6a-83b6a9268881","Type":"ContainerStarted","Data":"8477ca0eb9c10f485e2da3904594ad2b7f5085231cbf564783e2bec3960acad7"} Oct 30 16:34:26.922597 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:26.921803 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:26.944854 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:26.944826 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:27.699778 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:27.699685 5838 generic.go:356] "Generic (PLEG): container finished" podID="bf277455-983a-424b-bbdb-a33f3757ca8d" containerID="f482b6908622e2ebb3b16c5751540e2eec15e35346f33ef0326d3d045d83b467" exitCode=0 Oct 30 16:34:27.699980 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:27.699768 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" event={"ID":"bf277455-983a-424b-bbdb-a33f3757ca8d","Type":"ContainerDied","Data":"f482b6908622e2ebb3b16c5751540e2eec15e35346f33ef0326d3d045d83b467"} Oct 30 16:34:27.701315 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:27.701296 5838 generic.go:356] "Generic (PLEG): container finished" podID="3ce677bf-1156-4ede-ab6a-83b6a9268881" containerID="248464c2db6ecb65dbe2bf6bb79fd9c086ed6ad7f19e16149335281fd1cde605" exitCode=0 Oct 30 16:34:27.701411 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:27.701385 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" event={"ID":"3ce677bf-1156-4ede-ab6a-83b6a9268881","Type":"ContainerDied","Data":"248464c2db6ecb65dbe2bf6bb79fd9c086ed6ad7f19e16149335281fd1cde605"} Oct 30 16:34:27.712755 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:27.712731 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:34:28.883833 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.883807 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:28.887078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.887057 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:28.961543 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.961464 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce677bf-1156-4ede-ab6a-83b6a9268881-kube-api-access-bvh27\" (UniqueName: \"kubernetes.io/projected/3ce677bf-1156-4ede-ab6a-83b6a9268881-kube-api-access-bvh27\") pod \"3ce677bf-1156-4ede-ab6a-83b6a9268881\" (UID: \"3ce677bf-1156-4ede-ab6a-83b6a9268881\") " Oct 30 16:34:28.961543 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.961514 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-xks-ca\") pod \"bf277455-983a-424b-bbdb-a33f3757ca8d\" (UID: \"bf277455-983a-424b-bbdb-a33f3757ca8d\") " Oct 30 16:34:28.961742 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.961563 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-openshift-ca\") pod \"3ce677bf-1156-4ede-ab6a-83b6a9268881\" (UID: \"3ce677bf-1156-4ede-ab6a-83b6a9268881\") " Oct 30 16:34:28.961742 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.961613 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-xks-ca\") pod \"3ce677bf-1156-4ede-ab6a-83b6a9268881\" (UID: \"3ce677bf-1156-4ede-ab6a-83b6a9268881\") " Oct 30 16:34:28.961742 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.961665 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-openshift-ca\") pod \"bf277455-983a-424b-bbdb-a33f3757ca8d\" (UID: \"bf277455-983a-424b-bbdb-a33f3757ca8d\") " Oct 30 16:34:28.961742 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.961695 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf277455-983a-424b-bbdb-a33f3757ca8d-kube-api-access-fdck5\" (UniqueName: \"kubernetes.io/projected/bf277455-983a-424b-bbdb-a33f3757ca8d-kube-api-access-fdck5\") pod \"bf277455-983a-424b-bbdb-a33f3757ca8d\" (UID: \"bf277455-983a-424b-bbdb-a33f3757ca8d\") " Oct 30 16:34:28.962026 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.961955 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-openshift-ca" pod "3ce677bf-1156-4ede-ab6a-83b6a9268881" (UID: "3ce677bf-1156-4ede-ab6a-83b6a9268881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:34:28.962276 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.962104 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-openshift-ca" pod "bf277455-983a-424b-bbdb-a33f3757ca8d" (UID: "bf277455-983a-424b-bbdb-a33f3757ca8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:34:28.962276 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.962249 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:28.962276 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.962272 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:28.964120 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.964030 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-xks-ca" pod "3ce677bf-1156-4ede-ab6a-83b6a9268881" (UID: "3ce677bf-1156-4ede-ab6a-83b6a9268881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:34:28.964241 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.964173 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-xks-ca" pod "bf277455-983a-424b-bbdb-a33f3757ca8d" (UID: "bf277455-983a-424b-bbdb-a33f3757ca8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:34:28.964241 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.964211 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf277455-983a-424b-bbdb-a33f3757ca8d-kube-api-access-fdck5" pod "bf277455-983a-424b-bbdb-a33f3757ca8d" (UID: "bf277455-983a-424b-bbdb-a33f3757ca8d"). InnerVolumeSpecName "kube-api-access-fdck5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:34:28.964241 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:28.964238 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce677bf-1156-4ede-ab6a-83b6a9268881-kube-api-access-bvh27" pod "3ce677bf-1156-4ede-ab6a-83b6a9268881" (UID: "3ce677bf-1156-4ede-ab6a-83b6a9268881"). InnerVolumeSpecName "kube-api-access-bvh27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:34:29.063301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.063262 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvh27\" (UniqueName: \"kubernetes.io/projected/3ce677bf-1156-4ede-ab6a-83b6a9268881-kube-api-access-bvh27\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:29.063301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.063290 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf277455-983a-424b-bbdb-a33f3757ca8d-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:29.063301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.063300 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ce677bf-1156-4ede-ab6a-83b6a9268881-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:29.063301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.063311 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdck5\" (UniqueName: \"kubernetes.io/projected/bf277455-983a-424b-bbdb-a33f3757ca8d-kube-api-access-fdck5\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:29.204831 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.204792 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg"] Oct 30 16:34:29.205373 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.205357 5838 cpu_manager.go:414] "containerMap: removing container" podUID="bf277455-983a-424b-bbdb-a33f3757ca8d" containerName="revoke-keys" Oct 30 16:34:29.205429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.205374 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf277455-983a-424b-bbdb-a33f3757ca8d" containerName="revoke-keys" Oct 30 16:34:29.205429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.205397 5838 cpu_manager.go:414] "containerMap: removing container" podUID="3ce677bf-1156-4ede-ab6a-83b6a9268881" containerName="revoke-keys" Oct 30 16:34:29.205429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.205403 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce677bf-1156-4ede-ab6a-83b6a9268881" containerName="revoke-keys" Oct 30 16:34:29.205537 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.205481 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf277455-983a-424b-bbdb-a33f3757ca8d" containerName="revoke-keys" Oct 30 16:34:29.205537 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.205493 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce677bf-1156-4ede-ab6a-83b6a9268881" containerName="revoke-keys" Oct 30 16:34:29.209347 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.209309 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.228606 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.228569 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3e4f91\"/\"openshift-service-ca.crt\"" Oct 30 16:34:29.228823 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.228644 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3e4f91\"/\"premium-w2-3e4f91-kserve-self-signed-certs\"" Oct 30 16:34:29.228823 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.228708 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3e4f91\"/\"default-dockercfg-cd44t\"" Oct 30 16:34:29.228823 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.228758 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3e4f91\"/\"kube-root-ca.crt\"" Oct 30 16:34:29.229565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.229541 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg"] Oct 30 16:34:29.365446 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.365403 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g9kb\" (UniqueName: \"kubernetes.io/projected/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-kube-api-access-7g9kb\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.365651 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.365461 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-model-cache\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.365651 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.365578 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tls-certs\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.365651 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.365641 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tmp-dir\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.365801 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.365780 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-home\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.365838 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.365823 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-dshm\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.467120 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.467082 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g9kb\" (UniqueName: \"kubernetes.io/projected/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-kube-api-access-7g9kb\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.467296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.467129 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-model-cache\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.467296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.467162 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tls-certs\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.467296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.467196 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tmp-dir\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.467296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.467279 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-home\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.467558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.467305 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-dshm\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.467623 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.467607 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-model-cache\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.467881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.467857 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tmp-dir\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.467991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.467951 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-home\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.469537 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.469517 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-dshm\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.469687 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.469670 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tls-certs\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.489379 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.489351 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g9kb\" (UniqueName: \"kubernetes.io/projected/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-kube-api-access-7g9kb\") pod \"premium-w2-3e4f91-kserve-58664cbc64-bcvpg\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.524205 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.524169 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:29.716364 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.716335 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" Oct 30 16:34:29.716548 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.716373 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn" event={"ID":"3ce677bf-1156-4ede-ab6a-83b6a9268881","Type":"ContainerDied","Data":"8477ca0eb9c10f485e2da3904594ad2b7f5085231cbf564783e2bec3960acad7"} Oct 30 16:34:29.716548 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.716434 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8477ca0eb9c10f485e2da3904594ad2b7f5085231cbf564783e2bec3960acad7" Oct 30 16:34:29.717861 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.717799 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" event={"ID":"bf277455-983a-424b-bbdb-a33f3757ca8d","Type":"ContainerDied","Data":"d39ec6ad0be580b5ef77998958869197289bf48311b7452115fcd660257f1238"} Oct 30 16:34:29.717861 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.717840 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d39ec6ad0be580b5ef77998958869197289bf48311b7452115fcd660257f1238" Oct 30 16:34:29.717861 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.717803 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9" Oct 30 16:34:29.839404 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:29.839370 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg"] Oct 30 16:34:29.842847 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:34:29.842816 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod860612c0_368d_4e00_9b59_7b4cdbfe0c2d.slice/crio-aa1ed311683d10757fb3b6f69c5cbf3305485bed05dca88478c585ef178649f9 WatchSource:0}: Error finding container aa1ed311683d10757fb3b6f69c5cbf3305485bed05dca88478c585ef178649f9: Status 404 returned error can't find the container with id aa1ed311683d10757fb3b6f69c5cbf3305485bed05dca88478c585ef178649f9 Oct 30 16:34:30.725537 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:30.725498 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" event={"ID":"860612c0-368d-4e00-9b59-7b4cdbfe0c2d","Type":"ContainerStarted","Data":"b3b8f18939d2d512575a8e524bb9e589a890863cef4f0586045119c83cbe65d9"} Oct 30 16:34:30.725955 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:30.725551 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" event={"ID":"860612c0-368d-4e00-9b59-7b4cdbfe0c2d","Type":"ContainerStarted","Data":"aa1ed311683d10757fb3b6f69c5cbf3305485bed05dca88478c585ef178649f9"} Oct 30 16:34:30.766726 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:30.766665 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podStartSLOduration=1.766647224 podStartE2EDuration="1.766647224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:30.764404501 +0000 UTC m=+832.512993888" watchObservedRunningTime="2026-10-30 16:34:30.766647224 +0000 UTC m=+832.515236613" Oct 30 16:34:31.525198 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:31.525164 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:31.798043 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:34:31.797929 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:34:32.920698 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:32.920661 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29"] Oct 30 16:34:32.924875 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:32.924853 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:32.943869 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:32.943826 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29"] Oct 30 16:34:32.951884 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:32.951858 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:34:32.951884 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:32.951873 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:34:33.008920 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.008887 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzrlw\" (UniqueName: \"kubernetes.io/projected/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-kube-api-access-fzrlw\") pod \"maas-api-revoke-sub-6827378d-4lf29\" (UID: \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:33.009109 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.008930 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6827378d-4lf29\" (UID: \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:33.009148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.009108 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6827378d-4lf29\" (UID: \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:33.110018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.109963 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6827378d-4lf29\" (UID: \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:33.110189 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.110166 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzrlw\" (UniqueName: \"kubernetes.io/projected/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-kube-api-access-fzrlw\") pod \"maas-api-revoke-sub-6827378d-4lf29\" (UID: \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:33.110233 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.110208 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6827378d-4lf29\" (UID: \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:33.110700 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.110676 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6827378d-4lf29\" (UID: \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:33.112657 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.112633 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6827378d-4lf29\" (UID: \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:33.135924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.135897 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzrlw\" (UniqueName: \"kubernetes.io/projected/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-kube-api-access-fzrlw\") pod \"maas-api-revoke-sub-6827378d-4lf29\" (UID: \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:33.141326 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.141299 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv"] Oct 30 16:34:33.148101 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.148068 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.164873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.164848 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2e0784-openshift-default-dockercfg-4swtp\"" Oct 30 16:34:33.168738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.168716 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv"] Oct 30 16:34:33.239067 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.239031 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:33.312705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.312621 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-credential-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.312705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.312677 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318405fd-d020-4bb2-b582-44c2517639ef-istiod-ca-cert\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.312901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.312720 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-istio-token\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.312901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.312761 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-certs\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.312901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.312784 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318405fd-d020-4bb2-b582-44c2517639ef-istio-podinfo\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.312901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.312800 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2qh\" (UniqueName: \"kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-kube-api-access-tk2qh\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.312901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.312819 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-envoy\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.312901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.312835 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-data\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.312901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.312862 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.413994 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-certs\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.414041 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318405fd-d020-4bb2-b582-44c2517639ef-istio-podinfo\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414298 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.414152 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2qh\" (UniqueName: \"kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-kube-api-access-tk2qh\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414298 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.414201 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-envoy\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414298 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.414230 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-data\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414298 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.414280 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414509 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.414432 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-credential-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414509 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.414449 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-certs\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414509 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.414472 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318405fd-d020-4bb2-b582-44c2517639ef-istiod-ca-cert\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414649 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.414544 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-istio-token\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414726 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.414702 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-data\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.414715 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.414903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.414875 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-credential-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.415337 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.415315 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318405fd-d020-4bb2-b582-44c2517639ef-istiod-ca-cert\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.416488 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.416464 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-envoy\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.416696 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.416679 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318405fd-d020-4bb2-b582-44c2517639ef-istio-podinfo\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.437048 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.437021 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2qh\" (UniqueName: \"kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-kube-api-access-tk2qh\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.437221 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.437204 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-istio-token\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.463768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.463741 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:33.544603 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.544574 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29"] Oct 30 16:34:33.546207 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:34:33.546178 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3538f9ff_1ab1_4ac3_8fca_b79dc3106a2c.slice/crio-26c85a8f1dcab4d8ea75275dca53358b2fd21ac0d555123130091c400f65e9fa WatchSource:0}: Error finding container 26c85a8f1dcab4d8ea75275dca53358b2fd21ac0d555123130091c400f65e9fa: Status 404 returned error can't find the container with id 26c85a8f1dcab4d8ea75275dca53358b2fd21ac0d555123130091c400f65e9fa Oct 30 16:34:33.747541 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.747504 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" event={"ID":"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c","Type":"ContainerStarted","Data":"26c85a8f1dcab4d8ea75275dca53358b2fd21ac0d555123130091c400f65e9fa"} Oct 30 16:34:33.771006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.770960 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv"] Oct 30 16:34:33.773809 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:34:33.773778 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318405fd_d020_4bb2_b582_44c2517639ef.slice/crio-cfe09b8fd67631f114640d56c37292ee99102b73a59e249097f8517784f9f78b WatchSource:0}: Error finding container cfe09b8fd67631f114640d56c37292ee99102b73a59e249097f8517784f9f78b: Status 404 returned error can't find the container with id cfe09b8fd67631f114640d56c37292ee99102b73a59e249097f8517784f9f78b Oct 30 16:34:33.776294 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.776266 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:33.776379 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.776329 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:33.776379 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:33.776359 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:34.754780 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:34.754746 5838 generic.go:356] "Generic (PLEG): container finished" podID="3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c" containerID="89de5e902858c80419923098cb7441cdfff779c808f5fa2e920ca033c77a95c5" exitCode=0 Oct 30 16:34:34.755270 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:34.754835 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" event={"ID":"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c","Type":"ContainerDied","Data":"89de5e902858c80419923098cb7441cdfff779c808f5fa2e920ca033c77a95c5"} Oct 30 16:34:34.756589 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:34.756569 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" event={"ID":"318405fd-d020-4bb2-b582-44c2517639ef","Type":"ContainerStarted","Data":"eb6848775914395bc16c85d97e73a0a2731c4ae1dc687b057e39a048e6332122"} Oct 30 16:34:34.756704 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:34.756606 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" event={"ID":"318405fd-d020-4bb2-b582-44c2517639ef","Type":"ContainerStarted","Data":"cfe09b8fd67631f114640d56c37292ee99102b73a59e249097f8517784f9f78b"} Oct 30 16:34:34.841075 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:34.839552 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" podStartSLOduration=1.8395333520000001 podStartE2EDuration="1.839533352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:34.837671643 +0000 UTC m=+836.586261032" watchObservedRunningTime="2026-10-30 16:34:34.839533352 +0000 UTC m=+836.588122749" Oct 30 16:34:35.464901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:35.464871 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:35.466689 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:35.466662 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:34:35.466843 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:35.466716 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" podUID="318405fd-d020-4bb2-b582-44c2517639ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:34:35.949707 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:35.949676 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:36.056759 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.056666 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-kube-api-access-fzrlw\" (UniqueName: \"kubernetes.io/projected/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-kube-api-access-fzrlw\") pod \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\" (UID: \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\") " Oct 30 16:34:36.056937 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.056798 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-xks-ca\") pod \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\" (UID: \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\") " Oct 30 16:34:36.056937 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.056882 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-openshift-ca\") pod \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\" (UID: \"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c\") " Oct 30 16:34:36.057275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.057247 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-openshift-ca" pod "3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c" (UID: "3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:34:36.057362 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.057348 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:36.058810 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.058783 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-kube-api-access-fzrlw" pod "3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c" (UID: "3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c"). InnerVolumeSpecName "kube-api-access-fzrlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:34:36.058919 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.058809 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-xks-ca" pod "3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c" (UID: "3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:34:36.158844 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.158803 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:36.158844 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.158840 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzrlw\" (UniqueName: \"kubernetes.io/projected/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c-kube-api-access-fzrlw\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:36.464530 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.464495 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:34:36.464750 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.464551 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" podUID="318405fd-d020-4bb2-b582-44c2517639ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:34:36.770340 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.770308 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" Oct 30 16:34:36.770518 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.770332 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29" event={"ID":"3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c","Type":"ContainerDied","Data":"26c85a8f1dcab4d8ea75275dca53358b2fd21ac0d555123130091c400f65e9fa"} Oct 30 16:34:36.770518 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:36.770387 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26c85a8f1dcab4d8ea75275dca53358b2fd21ac0d555123130091c400f65e9fa" Oct 30 16:34:37.464223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.464183 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:34:37.464620 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.464236 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" podUID="318405fd-d020-4bb2-b582-44c2517639ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:34:37.767584 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.767548 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6"] Oct 30 16:34:37.768112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.768098 5838 cpu_manager.go:414] "containerMap: removing container" podUID="3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c" containerName="revoke-keys" Oct 30 16:34:37.768162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.768113 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c" containerName="revoke-keys" Oct 30 16:34:37.768236 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.768226 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c" containerName="revoke-keys" Oct 30 16:34:37.772135 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.772111 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:37.775232 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.775193 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc8fa13d-scgr6\" (UID: \"302d2f6e-678e-48b2-810d-1bdfb9d36342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:37.775369 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.775245 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc8fa13d-scgr6\" (UID: \"302d2f6e-678e-48b2-810d-1bdfb9d36342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:37.775439 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.775380 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxp9\" (UniqueName: \"kubernetes.io/projected/302d2f6e-678e-48b2-810d-1bdfb9d36342-kube-api-access-lfxp9\") pod \"maas-api-revoke-sub-cc8fa13d-scgr6\" (UID: \"302d2f6e-678e-48b2-810d-1bdfb9d36342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:37.790885 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.790853 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6"] Oct 30 16:34:37.801006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.798796 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:34:37.801006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.798816 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:34:37.875937 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.875901 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc8fa13d-scgr6\" (UID: \"302d2f6e-678e-48b2-810d-1bdfb9d36342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:37.876145 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.875945 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc8fa13d-scgr6\" (UID: \"302d2f6e-678e-48b2-810d-1bdfb9d36342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:37.876145 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.876029 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxp9\" (UniqueName: \"kubernetes.io/projected/302d2f6e-678e-48b2-810d-1bdfb9d36342-kube-api-access-lfxp9\") pod \"maas-api-revoke-sub-cc8fa13d-scgr6\" (UID: \"302d2f6e-678e-48b2-810d-1bdfb9d36342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:37.876557 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.876536 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc8fa13d-scgr6\" (UID: \"302d2f6e-678e-48b2-810d-1bdfb9d36342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:37.878312 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.878290 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc8fa13d-scgr6\" (UID: \"302d2f6e-678e-48b2-810d-1bdfb9d36342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:37.899279 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:37.899241 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxp9\" (UniqueName: \"kubernetes.io/projected/302d2f6e-678e-48b2-810d-1bdfb9d36342-kube-api-access-lfxp9\") pod \"maas-api-revoke-sub-cc8fa13d-scgr6\" (UID: \"302d2f6e-678e-48b2-810d-1bdfb9d36342\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:38.087223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.087144 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:38.411622 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.411014 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6"] Oct 30 16:34:38.442519 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:34:38.442466 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302d2f6e_678e_48b2_810d_1bdfb9d36342.slice/crio-f13f5335b843cb8b94d7e83930b154ad8327abf511c5fac72bffae65084e8fe3 WatchSource:0}: Error finding container f13f5335b843cb8b94d7e83930b154ad8327abf511c5fac72bffae65084e8fe3: Status 404 returned error can't find the container with id f13f5335b843cb8b94d7e83930b154ad8327abf511c5fac72bffae65084e8fe3 Oct 30 16:34:38.465577 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.465544 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:34:38.466030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.465597 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" podUID="318405fd-d020-4bb2-b582-44c2517639ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:34:38.785397 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.785361 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" event={"ID":"302d2f6e-678e-48b2-810d-1bdfb9d36342","Type":"ContainerStarted","Data":"f13f5335b843cb8b94d7e83930b154ad8327abf511c5fac72bffae65084e8fe3"} Oct 30 16:34:38.792116 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.792089 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb"] Oct 30 16:34:38.796074 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.796054 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:38.851905 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.851710 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb"] Oct 30 16:34:38.889324 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.889292 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf303e92-gtvbb\" (UID: \"906e3afd-5693-4dde-8904-5ef536742cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:38.889466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.889340 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf303e92-gtvbb\" (UID: \"906e3afd-5693-4dde-8904-5ef536742cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:38.889532 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.889509 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt7z4\" (UniqueName: \"kubernetes.io/projected/906e3afd-5693-4dde-8904-5ef536742cbb-kube-api-access-wt7z4\") pod \"maas-api-revoke-sub-bf303e92-gtvbb\" (UID: \"906e3afd-5693-4dde-8904-5ef536742cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:38.990921 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.990882 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf303e92-gtvbb\" (UID: \"906e3afd-5693-4dde-8904-5ef536742cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:38.991114 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.991066 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt7z4\" (UniqueName: \"kubernetes.io/projected/906e3afd-5693-4dde-8904-5ef536742cbb-kube-api-access-wt7z4\") pod \"maas-api-revoke-sub-bf303e92-gtvbb\" (UID: \"906e3afd-5693-4dde-8904-5ef536742cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:38.991199 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.991179 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf303e92-gtvbb\" (UID: \"906e3afd-5693-4dde-8904-5ef536742cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:38.991645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.991621 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf303e92-gtvbb\" (UID: \"906e3afd-5693-4dde-8904-5ef536742cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:38.993441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:38.993419 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf303e92-gtvbb\" (UID: \"906e3afd-5693-4dde-8904-5ef536742cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:39.013578 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.013554 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt7z4\" (UniqueName: \"kubernetes.io/projected/906e3afd-5693-4dde-8904-5ef536742cbb-kube-api-access-wt7z4\") pod \"maas-api-revoke-sub-bf303e92-gtvbb\" (UID: \"906e3afd-5693-4dde-8904-5ef536742cbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:39.065632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.065531 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:34:39.146608 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.146570 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:39.453812 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.453791 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb"] Oct 30 16:34:39.456061 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:34:39.456033 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906e3afd_5693_4dde_8904_5ef536742cbb.slice/crio-0980cb1a0e03dda37d67f96923ff906718b7047f7a88c86a0a0fe53db06b2aa4 WatchSource:0}: Error finding container 0980cb1a0e03dda37d67f96923ff906718b7047f7a88c86a0a0fe53db06b2aa4: Status 404 returned error can't find the container with id 0980cb1a0e03dda37d67f96923ff906718b7047f7a88c86a0a0fe53db06b2aa4 Oct 30 16:34:39.464751 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.464700 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:34:39.464872 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.464747 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" podUID="318405fd-d020-4bb2-b582-44c2517639ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:34:39.539034 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.535223 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:39.552414 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.552387 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:39.824353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.824295 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" event={"ID":"906e3afd-5693-4dde-8904-5ef536742cbb","Type":"ContainerStarted","Data":"d0dbe610ef6b1d5e1d1f1efb14bbe5f9a97b782aada55d569dab4828dbb9ec2e"} Oct 30 16:34:39.824568 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.824381 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" event={"ID":"906e3afd-5693-4dde-8904-5ef536742cbb","Type":"ContainerStarted","Data":"0980cb1a0e03dda37d67f96923ff906718b7047f7a88c86a0a0fe53db06b2aa4"} Oct 30 16:34:39.829443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.828187 5838 generic.go:356] "Generic (PLEG): container finished" podID="302d2f6e-678e-48b2-810d-1bdfb9d36342" containerID="abe068da2accb199d935f0cfc0c1fec21d4d40737353f6c049e448eca4cd3322" exitCode=0 Oct 30 16:34:39.829443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.828859 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" event={"ID":"302d2f6e-678e-48b2-810d-1bdfb9d36342","Type":"ContainerDied","Data":"abe068da2accb199d935f0cfc0c1fec21d4d40737353f6c049e448eca4cd3322"} Oct 30 16:34:39.840778 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.840753 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:34:39.866921 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:39.866870 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" podStartSLOduration=1.866856005 podStartE2EDuration="1.866856005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:39.866540789 +0000 UTC m=+841.615130176" watchObservedRunningTime="2026-10-30 16:34:39.866856005 +0000 UTC m=+841.615445393" Oct 30 16:34:40.840456 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:40.840407 5838 generic.go:356] "Generic (PLEG): container finished" podID="906e3afd-5693-4dde-8904-5ef536742cbb" containerID="d0dbe610ef6b1d5e1d1f1efb14bbe5f9a97b782aada55d569dab4828dbb9ec2e" exitCode=0 Oct 30 16:34:40.840953 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:40.840805 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" event={"ID":"906e3afd-5693-4dde-8904-5ef536742cbb","Type":"ContainerDied","Data":"d0dbe610ef6b1d5e1d1f1efb14bbe5f9a97b782aada55d569dab4828dbb9ec2e"} Oct 30 16:34:41.108013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.103886 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:41.140601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.140571 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-xks-ca\") pod \"302d2f6e-678e-48b2-810d-1bdfb9d36342\" (UID: \"302d2f6e-678e-48b2-810d-1bdfb9d36342\") " Oct 30 16:34:41.140812 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.140649 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302d2f6e-678e-48b2-810d-1bdfb9d36342-kube-api-access-lfxp9\" (UniqueName: \"kubernetes.io/projected/302d2f6e-678e-48b2-810d-1bdfb9d36342-kube-api-access-lfxp9\") pod \"302d2f6e-678e-48b2-810d-1bdfb9d36342\" (UID: \"302d2f6e-678e-48b2-810d-1bdfb9d36342\") " Oct 30 16:34:41.143684 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.143656 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302d2f6e-678e-48b2-810d-1bdfb9d36342-kube-api-access-lfxp9" pod "302d2f6e-678e-48b2-810d-1bdfb9d36342" (UID: "302d2f6e-678e-48b2-810d-1bdfb9d36342"). InnerVolumeSpecName "kube-api-access-lfxp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:34:41.143831 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.143720 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-xks-ca" pod "302d2f6e-678e-48b2-810d-1bdfb9d36342" (UID: "302d2f6e-678e-48b2-810d-1bdfb9d36342"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:34:41.241253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.241217 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-openshift-ca\") pod \"302d2f6e-678e-48b2-810d-1bdfb9d36342\" (UID: \"302d2f6e-678e-48b2-810d-1bdfb9d36342\") " Oct 30 16:34:41.241602 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.241586 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:41.241602 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.241603 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfxp9\" (UniqueName: \"kubernetes.io/projected/302d2f6e-678e-48b2-810d-1bdfb9d36342-kube-api-access-lfxp9\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:41.241742 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.241660 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-openshift-ca" pod "302d2f6e-678e-48b2-810d-1bdfb9d36342" (UID: "302d2f6e-678e-48b2-810d-1bdfb9d36342"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:34:41.332268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.332229 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:41.332765 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.332742 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:41.333608 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.333586 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:34:41.342521 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.342476 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302d2f6e-678e-48b2-810d-1bdfb9d36342-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:41.849705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.849671 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" Oct 30 16:34:41.850199 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.849705 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6" event={"ID":"302d2f6e-678e-48b2-810d-1bdfb9d36342","Type":"ContainerDied","Data":"f13f5335b843cb8b94d7e83930b154ad8327abf511c5fac72bffae65084e8fe3"} Oct 30 16:34:41.850199 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.849758 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13f5335b843cb8b94d7e83930b154ad8327abf511c5fac72bffae65084e8fe3" Oct 30 16:34:41.994884 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:41.994834 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:42.049944 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:42.049913 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-openshift-ca\") pod \"906e3afd-5693-4dde-8904-5ef536742cbb\" (UID: \"906e3afd-5693-4dde-8904-5ef536742cbb\") " Oct 30 16:34:42.050123 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:42.049964 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/906e3afd-5693-4dde-8904-5ef536742cbb-kube-api-access-wt7z4\" (UniqueName: \"kubernetes.io/projected/906e3afd-5693-4dde-8904-5ef536742cbb-kube-api-access-wt7z4\") pod \"906e3afd-5693-4dde-8904-5ef536742cbb\" (UID: \"906e3afd-5693-4dde-8904-5ef536742cbb\") " Oct 30 16:34:42.050123 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:42.050040 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-xks-ca\") pod \"906e3afd-5693-4dde-8904-5ef536742cbb\" (UID: \"906e3afd-5693-4dde-8904-5ef536742cbb\") " Oct 30 16:34:42.050361 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:42.050329 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-openshift-ca" pod "906e3afd-5693-4dde-8904-5ef536742cbb" (UID: "906e3afd-5693-4dde-8904-5ef536742cbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:34:42.050507 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:42.050490 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:42.052259 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:42.052231 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-xks-ca" pod "906e3afd-5693-4dde-8904-5ef536742cbb" (UID: "906e3afd-5693-4dde-8904-5ef536742cbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:34:42.052598 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:42.052571 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906e3afd-5693-4dde-8904-5ef536742cbb-kube-api-access-wt7z4" pod "906e3afd-5693-4dde-8904-5ef536742cbb" (UID: "906e3afd-5693-4dde-8904-5ef536742cbb"). InnerVolumeSpecName "kube-api-access-wt7z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:34:42.151200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:42.151118 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt7z4\" (UniqueName: \"kubernetes.io/projected/906e3afd-5693-4dde-8904-5ef536742cbb-kube-api-access-wt7z4\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:42.151200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:42.151148 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/906e3afd-5693-4dde-8904-5ef536742cbb-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:42.856717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:42.856685 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" Oct 30 16:34:42.856717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:42.856706 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb" event={"ID":"906e3afd-5693-4dde-8904-5ef536742cbb","Type":"ContainerDied","Data":"0980cb1a0e03dda37d67f96923ff906718b7047f7a88c86a0a0fe53db06b2aa4"} Oct 30 16:34:42.857201 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:42.856756 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0980cb1a0e03dda37d67f96923ff906718b7047f7a88c86a0a0fe53db06b2aa4" Oct 30 16:34:45.492293 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.492257 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv"] Oct 30 16:34:45.492818 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.492804 5838 cpu_manager.go:414] "containerMap: removing container" podUID="302d2f6e-678e-48b2-810d-1bdfb9d36342" containerName="revoke-keys" Oct 30 16:34:45.492862 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.492819 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302d2f6e-678e-48b2-810d-1bdfb9d36342" containerName="revoke-keys" Oct 30 16:34:45.492894 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.492862 5838 cpu_manager.go:414] "containerMap: removing container" podUID="906e3afd-5693-4dde-8904-5ef536742cbb" containerName="revoke-keys" Oct 30 16:34:45.492894 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.492868 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906e3afd-5693-4dde-8904-5ef536742cbb" containerName="revoke-keys" Oct 30 16:34:45.492954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.492936 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="906e3afd-5693-4dde-8904-5ef536742cbb" containerName="revoke-keys" Oct 30 16:34:45.492954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.492945 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="302d2f6e-678e-48b2-810d-1bdfb9d36342" containerName="revoke-keys" Oct 30 16:34:45.497148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.497129 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:45.512226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.512178 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:34:45.515763 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.515733 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv"] Oct 30 16:34:45.523510 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.523486 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:34:45.588812 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.588783 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25ab8ece-mdgfv\" (UID: \"fb757212-c7f6-47e0-9557-e45e7a630846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:45.588812 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.588817 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25ab8ece-mdgfv\" (UID: \"fb757212-c7f6-47e0-9557-e45e7a630846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:45.589017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.588995 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdqn5\" (UniqueName: \"kubernetes.io/projected/fb757212-c7f6-47e0-9557-e45e7a630846-kube-api-access-sdqn5\") pod \"maas-api-revoke-sub-25ab8ece-mdgfv\" (UID: \"fb757212-c7f6-47e0-9557-e45e7a630846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:45.690445 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.690379 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdqn5\" (UniqueName: \"kubernetes.io/projected/fb757212-c7f6-47e0-9557-e45e7a630846-kube-api-access-sdqn5\") pod \"maas-api-revoke-sub-25ab8ece-mdgfv\" (UID: \"fb757212-c7f6-47e0-9557-e45e7a630846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:45.690647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.690536 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25ab8ece-mdgfv\" (UID: \"fb757212-c7f6-47e0-9557-e45e7a630846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:45.690647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.690565 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25ab8ece-mdgfv\" (UID: \"fb757212-c7f6-47e0-9557-e45e7a630846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:45.691207 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.691186 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25ab8ece-mdgfv\" (UID: \"fb757212-c7f6-47e0-9557-e45e7a630846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:45.692958 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.692941 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25ab8ece-mdgfv\" (UID: \"fb757212-c7f6-47e0-9557-e45e7a630846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:45.713362 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.713338 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdqn5\" (UniqueName: \"kubernetes.io/projected/fb757212-c7f6-47e0-9557-e45e7a630846-kube-api-access-sdqn5\") pod \"maas-api-revoke-sub-25ab8ece-mdgfv\" (UID: \"fb757212-c7f6-47e0-9557-e45e7a630846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:45.810483 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:45.810393 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:46.112441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:46.112365 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv"] Oct 30 16:34:46.115933 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:34:46.115904 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb757212_c7f6_47e0_9557_e45e7a630846.slice/crio-ea96bdb9edbd90d1e67508a01d603f5758cb5bdf58d5cb5fdb6f25de2721a1d6 WatchSource:0}: Error finding container ea96bdb9edbd90d1e67508a01d603f5758cb5bdf58d5cb5fdb6f25de2721a1d6: Status 404 returned error can't find the container with id ea96bdb9edbd90d1e67508a01d603f5758cb5bdf58d5cb5fdb6f25de2721a1d6 Oct 30 16:34:46.895139 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:46.895095 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" event={"ID":"fb757212-c7f6-47e0-9557-e45e7a630846","Type":"ContainerStarted","Data":"36c40e0bd903c3df8108b7b44969ab276cb211e5337c30e6abd0c5456dff1f72"} Oct 30 16:34:46.895609 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:46.895152 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" event={"ID":"fb757212-c7f6-47e0-9557-e45e7a630846","Type":"ContainerStarted","Data":"ea96bdb9edbd90d1e67508a01d603f5758cb5bdf58d5cb5fdb6f25de2721a1d6"} Oct 30 16:34:46.937016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:46.936947 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" podStartSLOduration=1.936932444 podStartE2EDuration="1.936932444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:46.933488403 +0000 UTC m=+848.682077807" watchObservedRunningTime="2026-10-30 16:34:46.936932444 +0000 UTC m=+848.685521874" Oct 30 16:34:47.903288 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:47.903254 5838 generic.go:356] "Generic (PLEG): container finished" podID="fb757212-c7f6-47e0-9557-e45e7a630846" containerID="36c40e0bd903c3df8108b7b44969ab276cb211e5337c30e6abd0c5456dff1f72" exitCode=0 Oct 30 16:34:47.903704 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:47.903343 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" event={"ID":"fb757212-c7f6-47e0-9557-e45e7a630846","Type":"ContainerDied","Data":"36c40e0bd903c3df8108b7b44969ab276cb211e5337c30e6abd0c5456dff1f72"} Oct 30 16:34:48.885203 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.885167 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p"] Oct 30 16:34:48.889346 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.889322 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:48.919936 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.919891 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-063dcc-openshift-default-dockercfg-zggx2\"" Oct 30 16:34:48.924415 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.924386 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p"] Oct 30 16:34:48.927073 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.927047 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-certs\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:48.927205 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.927103 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:48.927205 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.927137 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-envoy\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:48.927205 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.927194 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-credential-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:48.927371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.927244 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgk2m\" (UniqueName: \"kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-kube-api-access-tgk2m\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:48.927371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.927273 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64654948-d621-4884-bb13-0501973a1d4f-istio-podinfo\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:48.927371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.927321 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-data\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:48.927514 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.927405 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-istio-token\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:48.927514 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:48.927464 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64654948-d621-4884-bb13-0501973a1d4f-istiod-ca-cert\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.028524 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.028485 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-envoy\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.028719 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.028555 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-credential-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.028719 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.028588 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgk2m\" (UniqueName: \"kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-kube-api-access-tgk2m\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.028719 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.028643 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64654948-d621-4884-bb13-0501973a1d4f-istio-podinfo\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.028897 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.028725 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-data\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.029260 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.029231 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-data\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.029429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.029333 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-istio-token\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.029429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.029393 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64654948-d621-4884-bb13-0501973a1d4f-istiod-ca-cert\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.029592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.029452 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-certs\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.029592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.029506 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.029831 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.029808 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.030098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.030078 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-certs\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.030263 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.030245 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64654948-d621-4884-bb13-0501973a1d4f-istiod-ca-cert\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.030458 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.030428 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-credential-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.031362 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.031315 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-envoy\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.032479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.032458 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64654948-d621-4884-bb13-0501973a1d4f-istio-podinfo\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.053892 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.053863 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgk2m\" (UniqueName: \"kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-kube-api-access-tgk2m\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.054401 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.054378 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-istio-token\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.074369 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.074339 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:49.130425 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.130388 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-openshift-ca\") pod \"fb757212-c7f6-47e0-9557-e45e7a630846\" (UID: \"fb757212-c7f6-47e0-9557-e45e7a630846\") " Oct 30 16:34:49.130588 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.130469 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-xks-ca\") pod \"fb757212-c7f6-47e0-9557-e45e7a630846\" (UID: \"fb757212-c7f6-47e0-9557-e45e7a630846\") " Oct 30 16:34:49.130665 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.130600 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb757212-c7f6-47e0-9557-e45e7a630846-kube-api-access-sdqn5\" (UniqueName: \"kubernetes.io/projected/fb757212-c7f6-47e0-9557-e45e7a630846-kube-api-access-sdqn5\") pod \"fb757212-c7f6-47e0-9557-e45e7a630846\" (UID: \"fb757212-c7f6-47e0-9557-e45e7a630846\") " Oct 30 16:34:49.130818 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.130779 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-openshift-ca" pod "fb757212-c7f6-47e0-9557-e45e7a630846" (UID: "fb757212-c7f6-47e0-9557-e45e7a630846"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:34:49.131079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.131059 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:49.132733 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.132705 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-xks-ca" pod "fb757212-c7f6-47e0-9557-e45e7a630846" (UID: "fb757212-c7f6-47e0-9557-e45e7a630846"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:34:49.132829 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.132722 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb757212-c7f6-47e0-9557-e45e7a630846-kube-api-access-sdqn5" pod "fb757212-c7f6-47e0-9557-e45e7a630846" (UID: "fb757212-c7f6-47e0-9557-e45e7a630846"). InnerVolumeSpecName "kube-api-access-sdqn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:34:49.205161 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.202401 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8"] Oct 30 16:34:49.205161 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.203249 5838 cpu_manager.go:414] "containerMap: removing container" podUID="fb757212-c7f6-47e0-9557-e45e7a630846" containerName="revoke-keys" Oct 30 16:34:49.205161 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.203270 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb757212-c7f6-47e0-9557-e45e7a630846" containerName="revoke-keys" Oct 30 16:34:49.205161 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.203399 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb757212-c7f6-47e0-9557-e45e7a630846" containerName="revoke-keys" Oct 30 16:34:49.205889 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.205706 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:49.207755 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.207698 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:49.227991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.227954 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8"] Oct 30 16:34:49.232678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.232599 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkpnc\" (UniqueName: \"kubernetes.io/projected/eeaff967-55d3-45bb-b745-ac0a320ece42-kube-api-access-mkpnc\") pod \"maas-api-revoke-sub-59d6f232-blnv8\" (UID: \"eeaff967-55d3-45bb-b745-ac0a320ece42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:49.232795 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.232677 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d6f232-blnv8\" (UID: \"eeaff967-55d3-45bb-b745-ac0a320ece42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:49.232795 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.232756 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d6f232-blnv8\" (UID: \"eeaff967-55d3-45bb-b745-ac0a320ece42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:49.232983 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.232825 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdqn5\" (UniqueName: \"kubernetes.io/projected/fb757212-c7f6-47e0-9557-e45e7a630846-kube-api-access-sdqn5\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:49.232983 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.232842 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb757212-c7f6-47e0-9557-e45e7a630846-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:49.334213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.334176 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d6f232-blnv8\" (UID: \"eeaff967-55d3-45bb-b745-ac0a320ece42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:49.334382 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.334247 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d6f232-blnv8\" (UID: \"eeaff967-55d3-45bb-b745-ac0a320ece42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:49.334382 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.334378 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkpnc\" (UniqueName: \"kubernetes.io/projected/eeaff967-55d3-45bb-b745-ac0a320ece42-kube-api-access-mkpnc\") pod \"maas-api-revoke-sub-59d6f232-blnv8\" (UID: \"eeaff967-55d3-45bb-b745-ac0a320ece42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:49.334922 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.334898 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d6f232-blnv8\" (UID: \"eeaff967-55d3-45bb-b745-ac0a320ece42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:49.336663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.336643 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d6f232-blnv8\" (UID: \"eeaff967-55d3-45bb-b745-ac0a320ece42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:49.357083 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.357057 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkpnc\" (UniqueName: \"kubernetes.io/projected/eeaff967-55d3-45bb-b745-ac0a320ece42-kube-api-access-mkpnc\") pod \"maas-api-revoke-sub-59d6f232-blnv8\" (UID: \"eeaff967-55d3-45bb-b745-ac0a320ece42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:49.525228 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.525188 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:49.537740 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.537429 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p"] Oct 30 16:34:49.541487 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:34:49.541388 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64654948_d621_4884_bb13_0501973a1d4f.slice/crio-d576f688427221b093498c3785cdf4748f372d20b89daafcb270fda69a6d29a8 WatchSource:0}: Error finding container d576f688427221b093498c3785cdf4748f372d20b89daafcb270fda69a6d29a8: Status 404 returned error can't find the container with id d576f688427221b093498c3785cdf4748f372d20b89daafcb270fda69a6d29a8 Oct 30 16:34:49.544080 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.544046 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:49.544185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.544129 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:49.544249 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.544193 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:34:49.838953 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.838927 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8"] Oct 30 16:34:49.842365 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:34:49.842337 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeaff967_55d3_45bb_b745_ac0a320ece42.slice/crio-85df63c313c9a3d2677e176db1a05c1c98f8ce76fee2a7bea63ff398195b13ea WatchSource:0}: Error finding container 85df63c313c9a3d2677e176db1a05c1c98f8ce76fee2a7bea63ff398195b13ea: Status 404 returned error can't find the container with id 85df63c313c9a3d2677e176db1a05c1c98f8ce76fee2a7bea63ff398195b13ea Oct 30 16:34:49.920777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.920730 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" event={"ID":"64654948-d621-4884-bb13-0501973a1d4f","Type":"ContainerStarted","Data":"2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a"} Oct 30 16:34:49.921215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.920797 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" event={"ID":"64654948-d621-4884-bb13-0501973a1d4f","Type":"ContainerStarted","Data":"d576f688427221b093498c3785cdf4748f372d20b89daafcb270fda69a6d29a8"} Oct 30 16:34:49.922302 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.922273 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" event={"ID":"eeaff967-55d3-45bb-b745-ac0a320ece42","Type":"ContainerStarted","Data":"85df63c313c9a3d2677e176db1a05c1c98f8ce76fee2a7bea63ff398195b13ea"} Oct 30 16:34:49.923685 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.923669 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" Oct 30 16:34:49.923685 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.923677 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv" event={"ID":"fb757212-c7f6-47e0-9557-e45e7a630846","Type":"ContainerDied","Data":"ea96bdb9edbd90d1e67508a01d603f5758cb5bdf58d5cb5fdb6f25de2721a1d6"} Oct 30 16:34:49.923832 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.923716 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea96bdb9edbd90d1e67508a01d603f5758cb5bdf58d5cb5fdb6f25de2721a1d6" Oct 30 16:34:49.968953 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:49.968891 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" podStartSLOduration=1.968875584 podStartE2EDuration="1.968875584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:49.965547037 +0000 UTC m=+851.714136444" watchObservedRunningTime="2026-10-30 16:34:49.968875584 +0000 UTC m=+851.717464968" Oct 30 16:34:50.209777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:50.209691 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:50.210963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:50.210938 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:34:50.211094 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:50.210999 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" podUID="64654948-d621-4884-bb13-0501973a1d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:34:50.945420 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:50.945388 5838 generic.go:356] "Generic (PLEG): container finished" podID="eeaff967-55d3-45bb-b745-ac0a320ece42" containerID="fa48448a73bb9556de3bfaf09ea2a81d217e143ccf675af91c42a49f0ae8ae51" exitCode=0 Oct 30 16:34:50.945782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:50.945471 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" event={"ID":"eeaff967-55d3-45bb-b745-ac0a320ece42","Type":"ContainerDied","Data":"fa48448a73bb9556de3bfaf09ea2a81d217e143ccf675af91c42a49f0ae8ae51"} Oct 30 16:34:51.209932 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:51.209840 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:34:51.209932 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:51.209901 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" podUID="64654948-d621-4884-bb13-0501973a1d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:34:52.112351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.112319 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:52.174323 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.174290 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-openshift-ca\") pod \"eeaff967-55d3-45bb-b745-ac0a320ece42\" (UID: \"eeaff967-55d3-45bb-b745-ac0a320ece42\") " Oct 30 16:34:52.174523 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.174337 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeaff967-55d3-45bb-b745-ac0a320ece42-kube-api-access-mkpnc\" (UniqueName: \"kubernetes.io/projected/eeaff967-55d3-45bb-b745-ac0a320ece42-kube-api-access-mkpnc\") pod \"eeaff967-55d3-45bb-b745-ac0a320ece42\" (UID: \"eeaff967-55d3-45bb-b745-ac0a320ece42\") " Oct 30 16:34:52.174523 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.174375 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-xks-ca\") pod \"eeaff967-55d3-45bb-b745-ac0a320ece42\" (UID: \"eeaff967-55d3-45bb-b745-ac0a320ece42\") " Oct 30 16:34:52.175497 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.175464 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-openshift-ca" pod "eeaff967-55d3-45bb-b745-ac0a320ece42" (UID: "eeaff967-55d3-45bb-b745-ac0a320ece42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:34:52.175986 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.175946 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:52.177224 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.177182 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-xks-ca" pod "eeaff967-55d3-45bb-b745-ac0a320ece42" (UID: "eeaff967-55d3-45bb-b745-ac0a320ece42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:34:52.177802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.177767 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeaff967-55d3-45bb-b745-ac0a320ece42-kube-api-access-mkpnc" pod "eeaff967-55d3-45bb-b745-ac0a320ece42" (UID: "eeaff967-55d3-45bb-b745-ac0a320ece42"). InnerVolumeSpecName "kube-api-access-mkpnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:34:52.210303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.210227 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:34:52.210303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.210283 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" podUID="64654948-d621-4884-bb13-0501973a1d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:34:52.276953 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.276920 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkpnc\" (UniqueName: \"kubernetes.io/projected/eeaff967-55d3-45bb-b745-ac0a320ece42-kube-api-access-mkpnc\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:52.276953 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.276952 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeaff967-55d3-45bb-b745-ac0a320ece42-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:34:52.958949 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.958918 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" Oct 30 16:34:52.958949 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.958945 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8" event={"ID":"eeaff967-55d3-45bb-b745-ac0a320ece42","Type":"ContainerDied","Data":"85df63c313c9a3d2677e176db1a05c1c98f8ce76fee2a7bea63ff398195b13ea"} Oct 30 16:34:52.959181 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:52.959006 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85df63c313c9a3d2677e176db1a05c1c98f8ce76fee2a7bea63ff398195b13ea" Oct 30 16:34:53.210568 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:53.210484 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:34:53.210568 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:53.210546 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" podUID="64654948-d621-4884-bb13-0501973a1d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:34:54.209942 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:54.209907 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:34:54.210157 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:54.209992 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" podUID="64654948-d621-4884-bb13-0501973a1d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:34:55.209737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:55.209690 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:34:55.210776 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:55.210320 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" podUID="64654948-d621-4884-bb13-0501973a1d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:34:56.210317 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:56.210281 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:34:56.210794 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:56.210342 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" podUID="64654948-d621-4884-bb13-0501973a1d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:34:57.464775 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:57.464744 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:57.465299 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:57.465246 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:34:57.467285 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:34:57.467263 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:35:00.403556 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.403518 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j"] Oct 30 16:35:00.404115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.404098 5838 cpu_manager.go:414] "containerMap: removing container" podUID="eeaff967-55d3-45bb-b745-ac0a320ece42" containerName="revoke-keys" Oct 30 16:35:00.404176 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.404117 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeaff967-55d3-45bb-b745-ac0a320ece42" containerName="revoke-keys" Oct 30 16:35:00.404251 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.404238 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeaff967-55d3-45bb-b745-ac0a320ece42" containerName="revoke-keys" Oct 30 16:35:00.411776 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.411753 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:00.429312 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.429278 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j"] Oct 30 16:35:00.438017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.437104 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:35:00.438017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.437234 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:35:00.542358 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.542327 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsvjp\" (UniqueName: \"kubernetes.io/projected/c5d35e40-a13d-4e71-ab84-4ee846e50241-kube-api-access-xsvjp\") pod \"maas-api-revoke-sub-81d33ab8-v7v9j\" (UID: \"c5d35e40-a13d-4e71-ab84-4ee846e50241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:00.542502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.542377 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81d33ab8-v7v9j\" (UID: \"c5d35e40-a13d-4e71-ab84-4ee846e50241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:00.542502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.542411 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81d33ab8-v7v9j\" (UID: \"c5d35e40-a13d-4e71-ab84-4ee846e50241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:00.643835 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.643780 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsvjp\" (UniqueName: \"kubernetes.io/projected/c5d35e40-a13d-4e71-ab84-4ee846e50241-kube-api-access-xsvjp\") pod \"maas-api-revoke-sub-81d33ab8-v7v9j\" (UID: \"c5d35e40-a13d-4e71-ab84-4ee846e50241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:00.644058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.643876 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81d33ab8-v7v9j\" (UID: \"c5d35e40-a13d-4e71-ab84-4ee846e50241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:00.644058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.643920 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81d33ab8-v7v9j\" (UID: \"c5d35e40-a13d-4e71-ab84-4ee846e50241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:00.644714 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.644686 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81d33ab8-v7v9j\" (UID: \"c5d35e40-a13d-4e71-ab84-4ee846e50241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:00.646383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.646352 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81d33ab8-v7v9j\" (UID: \"c5d35e40-a13d-4e71-ab84-4ee846e50241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:00.667470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.667388 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsvjp\" (UniqueName: \"kubernetes.io/projected/c5d35e40-a13d-4e71-ab84-4ee846e50241-kube-api-access-xsvjp\") pod \"maas-api-revoke-sub-81d33ab8-v7v9j\" (UID: \"c5d35e40-a13d-4e71-ab84-4ee846e50241\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:00.728186 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:00.728152 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:01.039905 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.039878 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j"] Oct 30 16:35:01.043454 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:35:01.043419 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d35e40_a13d_4e71_ab84_4ee846e50241.slice/crio-fee7bbbcf300b3a8d6cce3391b97ec229cacfd089a20f77db9e66f7b31d75a4f WatchSource:0}: Error finding container fee7bbbcf300b3a8d6cce3391b97ec229cacfd089a20f77db9e66f7b31d75a4f: Status 404 returned error can't find the container with id fee7bbbcf300b3a8d6cce3391b97ec229cacfd089a20f77db9e66f7b31d75a4f Oct 30 16:35:01.067291 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.067264 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb"] Oct 30 16:35:01.072896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.072873 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:01.100851 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.100816 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb"] Oct 30 16:35:01.254141 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.254101 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-729178ae-lpsdb\" (UID: \"f408729b-6480-4c89-8944-f9a7e1ab5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:01.254330 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.254162 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf8vq\" (UniqueName: \"kubernetes.io/projected/f408729b-6480-4c89-8944-f9a7e1ab5736-kube-api-access-zf8vq\") pod \"maas-api-revoke-sub-729178ae-lpsdb\" (UID: \"f408729b-6480-4c89-8944-f9a7e1ab5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:01.254330 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.254233 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-729178ae-lpsdb\" (UID: \"f408729b-6480-4c89-8944-f9a7e1ab5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:01.355869 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.355834 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-729178ae-lpsdb\" (UID: \"f408729b-6480-4c89-8944-f9a7e1ab5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:01.356043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.355898 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8vq\" (UniqueName: \"kubernetes.io/projected/f408729b-6480-4c89-8944-f9a7e1ab5736-kube-api-access-zf8vq\") pod \"maas-api-revoke-sub-729178ae-lpsdb\" (UID: \"f408729b-6480-4c89-8944-f9a7e1ab5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:01.356043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.355926 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-729178ae-lpsdb\" (UID: \"f408729b-6480-4c89-8944-f9a7e1ab5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:01.356657 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.356633 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-729178ae-lpsdb\" (UID: \"f408729b-6480-4c89-8944-f9a7e1ab5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:01.358322 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.358296 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-729178ae-lpsdb\" (UID: \"f408729b-6480-4c89-8944-f9a7e1ab5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:01.383293 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.383264 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf8vq\" (UniqueName: \"kubernetes.io/projected/f408729b-6480-4c89-8944-f9a7e1ab5736-kube-api-access-zf8vq\") pod \"maas-api-revoke-sub-729178ae-lpsdb\" (UID: \"f408729b-6480-4c89-8944-f9a7e1ab5736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:01.399936 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.399902 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:01.709708 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:01.709615 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb"] Oct 30 16:35:01.719855 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:35:01.719808 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf408729b_6480_4c89_8944_f9a7e1ab5736.slice/crio-200675ff9575e7836c7e2540eeacf5e5d48d01149aca9b2f7af2a42f6f9a70cb WatchSource:0}: Error finding container 200675ff9575e7836c7e2540eeacf5e5d48d01149aca9b2f7af2a42f6f9a70cb: Status 404 returned error can't find the container with id 200675ff9575e7836c7e2540eeacf5e5d48d01149aca9b2f7af2a42f6f9a70cb Oct 30 16:35:01.827238 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:35:01.827190 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:35:02.037460 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:02.037424 5838 generic.go:356] "Generic (PLEG): container finished" podID="c5d35e40-a13d-4e71-ab84-4ee846e50241" containerID="961b28dda32ad29a2f3d5468f507e7d481e93f721917cbd6dd36e35efb977e51" exitCode=0 Oct 30 16:35:02.037632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:02.037507 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" event={"ID":"c5d35e40-a13d-4e71-ab84-4ee846e50241","Type":"ContainerDied","Data":"961b28dda32ad29a2f3d5468f507e7d481e93f721917cbd6dd36e35efb977e51"} Oct 30 16:35:02.037632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:02.037567 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" event={"ID":"c5d35e40-a13d-4e71-ab84-4ee846e50241","Type":"ContainerStarted","Data":"fee7bbbcf300b3a8d6cce3391b97ec229cacfd089a20f77db9e66f7b31d75a4f"} Oct 30 16:35:02.039234 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:02.039210 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" event={"ID":"f408729b-6480-4c89-8944-f9a7e1ab5736","Type":"ContainerStarted","Data":"033ff8103126d53c0641dec71490c0e4a656ebca34442fa6ffd235f70dd49e06"} Oct 30 16:35:02.039338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:02.039248 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" event={"ID":"f408729b-6480-4c89-8944-f9a7e1ab5736","Type":"ContainerStarted","Data":"200675ff9575e7836c7e2540eeacf5e5d48d01149aca9b2f7af2a42f6f9a70cb"} Oct 30 16:35:02.181486 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:02.181429 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" podStartSLOduration=1.18141358 podStartE2EDuration="1.18141358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:02.178643761 +0000 UTC m=+863.927233148" watchObservedRunningTime="2026-10-30 16:35:02.18141358 +0000 UTC m=+863.930002966" Oct 30 16:35:03.047582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.047548 5838 generic.go:356] "Generic (PLEG): container finished" podID="f408729b-6480-4c89-8944-f9a7e1ab5736" containerID="033ff8103126d53c0641dec71490c0e4a656ebca34442fa6ffd235f70dd49e06" exitCode=0 Oct 30 16:35:03.048053 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.047620 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" event={"ID":"f408729b-6480-4c89-8944-f9a7e1ab5736","Type":"ContainerDied","Data":"033ff8103126d53c0641dec71490c0e4a656ebca34442fa6ffd235f70dd49e06"} Oct 30 16:35:03.221701 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.221664 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:03.381615 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.381527 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-xks-ca\") pod \"c5d35e40-a13d-4e71-ab84-4ee846e50241\" (UID: \"c5d35e40-a13d-4e71-ab84-4ee846e50241\") " Oct 30 16:35:03.381615 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.381572 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d35e40-a13d-4e71-ab84-4ee846e50241-kube-api-access-xsvjp\" (UniqueName: \"kubernetes.io/projected/c5d35e40-a13d-4e71-ab84-4ee846e50241-kube-api-access-xsvjp\") pod \"c5d35e40-a13d-4e71-ab84-4ee846e50241\" (UID: \"c5d35e40-a13d-4e71-ab84-4ee846e50241\") " Oct 30 16:35:03.381809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.381722 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-openshift-ca\") pod \"c5d35e40-a13d-4e71-ab84-4ee846e50241\" (UID: \"c5d35e40-a13d-4e71-ab84-4ee846e50241\") " Oct 30 16:35:03.382149 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.382119 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-openshift-ca" pod "c5d35e40-a13d-4e71-ab84-4ee846e50241" (UID: "c5d35e40-a13d-4e71-ab84-4ee846e50241"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:03.382271 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.382218 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:03.383860 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.383834 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-xks-ca" pod "c5d35e40-a13d-4e71-ab84-4ee846e50241" (UID: "c5d35e40-a13d-4e71-ab84-4ee846e50241"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:03.383937 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.383907 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d35e40-a13d-4e71-ab84-4ee846e50241-kube-api-access-xsvjp" pod "c5d35e40-a13d-4e71-ab84-4ee846e50241" (UID: "c5d35e40-a13d-4e71-ab84-4ee846e50241"). InnerVolumeSpecName "kube-api-access-xsvjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:03.483479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.483443 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5d35e40-a13d-4e71-ab84-4ee846e50241-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:03.483479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.483473 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsvjp\" (UniqueName: \"kubernetes.io/projected/c5d35e40-a13d-4e71-ab84-4ee846e50241-kube-api-access-xsvjp\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:03.521593 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.521557 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls"] Oct 30 16:35:03.522150 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.522129 5838 cpu_manager.go:414] "containerMap: removing container" podUID="c5d35e40-a13d-4e71-ab84-4ee846e50241" containerName="revoke-keys" Oct 30 16:35:03.522150 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.522146 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d35e40-a13d-4e71-ab84-4ee846e50241" containerName="revoke-keys" Oct 30 16:35:03.522331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.522237 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d35e40-a13d-4e71-ab84-4ee846e50241" containerName="revoke-keys" Oct 30 16:35:03.525960 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.525940 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:03.549259 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.549202 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls"] Oct 30 16:35:03.685409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.685317 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-def4440f-r4bls\" (UID: \"a9ddd201-b315-49c5-83bf-6cacc3890c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:03.685409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.685355 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-def4440f-r4bls\" (UID: \"a9ddd201-b315-49c5-83bf-6cacc3890c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:03.685620 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.685506 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcw5d\" (UniqueName: \"kubernetes.io/projected/a9ddd201-b315-49c5-83bf-6cacc3890c13-kube-api-access-gcw5d\") pod \"maas-api-revoke-sub-def4440f-r4bls\" (UID: \"a9ddd201-b315-49c5-83bf-6cacc3890c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:03.786752 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.786715 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-def4440f-r4bls\" (UID: \"a9ddd201-b315-49c5-83bf-6cacc3890c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:03.786752 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.786753 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-def4440f-r4bls\" (UID: \"a9ddd201-b315-49c5-83bf-6cacc3890c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:03.787047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.786834 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcw5d\" (UniqueName: \"kubernetes.io/projected/a9ddd201-b315-49c5-83bf-6cacc3890c13-kube-api-access-gcw5d\") pod \"maas-api-revoke-sub-def4440f-r4bls\" (UID: \"a9ddd201-b315-49c5-83bf-6cacc3890c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:03.787457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.787426 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-def4440f-r4bls\" (UID: \"a9ddd201-b315-49c5-83bf-6cacc3890c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:03.789266 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.789240 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-def4440f-r4bls\" (UID: \"a9ddd201-b315-49c5-83bf-6cacc3890c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:03.805725 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.805686 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch"] Oct 30 16:35:03.810413 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.810386 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:03.814514 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.814489 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcw5d\" (UniqueName: \"kubernetes.io/projected/a9ddd201-b315-49c5-83bf-6cacc3890c13-kube-api-access-gcw5d\") pod \"maas-api-revoke-sub-def4440f-r4bls\" (UID: \"a9ddd201-b315-49c5-83bf-6cacc3890c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:03.824960 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.824925 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3e4f91-openshift-default-dockercfg-zfw6z\"" Oct 30 16:35:03.840355 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.840319 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch"] Oct 30 16:35:03.848199 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.848170 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:03.989768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.989730 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-credential-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:03.990012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.989791 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-token\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:03.990012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.989923 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-data\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:03.990012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.989959 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-certs\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:03.990214 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.990038 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-envoy\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:03.990214 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.990089 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:03.990214 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.990108 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istiod-ca-cert\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:03.990214 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.990208 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-podinfo\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:03.990440 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:03.990271 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6zgt\" (UniqueName: \"kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-kube-api-access-t6zgt\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.056006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.055946 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" Oct 30 16:35:04.056391 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.056010 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j" event={"ID":"c5d35e40-a13d-4e71-ab84-4ee846e50241","Type":"ContainerDied","Data":"fee7bbbcf300b3a8d6cce3391b97ec229cacfd089a20f77db9e66f7b31d75a4f"} Oct 30 16:35:04.056391 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.056060 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fee7bbbcf300b3a8d6cce3391b97ec229cacfd089a20f77db9e66f7b31d75a4f" Oct 30 16:35:04.091324 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.091285 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-credential-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.091539 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.091349 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-token\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.091539 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.091417 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-data\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.091539 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.091443 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-certs\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.091539 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.091467 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-envoy\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.091828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.091614 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.091828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.091659 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istiod-ca-cert\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.091828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.091696 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-podinfo\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.091828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.091784 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-credential-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.091828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.091812 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6zgt\" (UniqueName: \"kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-kube-api-access-t6zgt\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.092139 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.091898 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-data\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.092321 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.092302 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.092428 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.092407 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-certs\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.092852 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.092824 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istiod-ca-cert\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.093956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.093925 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-envoy\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.094274 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.094257 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-podinfo\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.115024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.114957 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-token\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.115578 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.115546 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6zgt\" (UniqueName: \"kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-kube-api-access-t6zgt\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.140464 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.140428 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:04.173981 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.173919 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls"] Oct 30 16:35:04.199125 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.199053 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:04.299943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.299908 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f408729b-6480-4c89-8944-f9a7e1ab5736-kube-api-access-zf8vq\" (UniqueName: \"kubernetes.io/projected/f408729b-6480-4c89-8944-f9a7e1ab5736-kube-api-access-zf8vq\") pod \"f408729b-6480-4c89-8944-f9a7e1ab5736\" (UID: \"f408729b-6480-4c89-8944-f9a7e1ab5736\") " Oct 30 16:35:04.300134 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.300060 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-openshift-ca\") pod \"f408729b-6480-4c89-8944-f9a7e1ab5736\" (UID: \"f408729b-6480-4c89-8944-f9a7e1ab5736\") " Oct 30 16:35:04.300134 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.300112 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-xks-ca\") pod \"f408729b-6480-4c89-8944-f9a7e1ab5736\" (UID: \"f408729b-6480-4c89-8944-f9a7e1ab5736\") " Oct 30 16:35:04.300418 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.300385 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-openshift-ca" pod "f408729b-6480-4c89-8944-f9a7e1ab5736" (UID: "f408729b-6480-4c89-8944-f9a7e1ab5736"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:04.302079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.302050 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-xks-ca" pod "f408729b-6480-4c89-8944-f9a7e1ab5736" (UID: "f408729b-6480-4c89-8944-f9a7e1ab5736"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:04.302179 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.302157 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f408729b-6480-4c89-8944-f9a7e1ab5736-kube-api-access-zf8vq" pod "f408729b-6480-4c89-8944-f9a7e1ab5736" (UID: "f408729b-6480-4c89-8944-f9a7e1ab5736"). InnerVolumeSpecName "kube-api-access-zf8vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:04.405850 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.405806 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf8vq\" (UniqueName: \"kubernetes.io/projected/f408729b-6480-4c89-8944-f9a7e1ab5736-kube-api-access-zf8vq\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:04.405850 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.405851 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:04.406120 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.405872 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f408729b-6480-4c89-8944-f9a7e1ab5736-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:04.468515 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.468487 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch"] Oct 30 16:35:04.470849 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:35:04.470822 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14848f01_2514_46ed_b1e3_cfbbca9ab49d.slice/crio-3cfa0bd4a52f4e2be459fa619d893e41dc3e0fbb1c042d8ff179366c444213f4 WatchSource:0}: Error finding container 3cfa0bd4a52f4e2be459fa619d893e41dc3e0fbb1c042d8ff179366c444213f4: Status 404 returned error can't find the container with id 3cfa0bd4a52f4e2be459fa619d893e41dc3e0fbb1c042d8ff179366c444213f4 Oct 30 16:35:04.473422 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.473383 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:04.473492 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.473467 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:04.473536 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:04.473506 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:05.065124 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:05.065084 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" event={"ID":"14848f01-2514-46ed-b1e3-cfbbca9ab49d","Type":"ContainerStarted","Data":"810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191"} Oct 30 16:35:05.065582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:05.065156 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" event={"ID":"14848f01-2514-46ed-b1e3-cfbbca9ab49d","Type":"ContainerStarted","Data":"3cfa0bd4a52f4e2be459fa619d893e41dc3e0fbb1c042d8ff179366c444213f4"} Oct 30 16:35:05.066785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:05.066761 5838 generic.go:356] "Generic (PLEG): container finished" podID="a9ddd201-b315-49c5-83bf-6cacc3890c13" containerID="3f413fbe8428acb53a7bc2747e8435378a0efc099acb1d06ebe2400cfaeef290" exitCode=0 Oct 30 16:35:05.066894 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:05.066844 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" event={"ID":"a9ddd201-b315-49c5-83bf-6cacc3890c13","Type":"ContainerDied","Data":"3f413fbe8428acb53a7bc2747e8435378a0efc099acb1d06ebe2400cfaeef290"} Oct 30 16:35:05.066894 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:05.066887 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" event={"ID":"a9ddd201-b315-49c5-83bf-6cacc3890c13","Type":"ContainerStarted","Data":"c662466abd3ea25a84ce1ec2fce8efb83dea470a15d77273fca3e4e455378dd3"} Oct 30 16:35:05.068177 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:05.068157 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" event={"ID":"f408729b-6480-4c89-8944-f9a7e1ab5736","Type":"ContainerDied","Data":"200675ff9575e7836c7e2540eeacf5e5d48d01149aca9b2f7af2a42f6f9a70cb"} Oct 30 16:35:05.068293 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:05.068188 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb" Oct 30 16:35:05.068293 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:05.068191 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="200675ff9575e7836c7e2540eeacf5e5d48d01149aca9b2f7af2a42f6f9a70cb" Oct 30 16:35:05.140803 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:05.140745 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" podStartSLOduration=2.140730298 podStartE2EDuration="2.140730298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:05.111528222 +0000 UTC m=+866.860117628" watchObservedRunningTime="2026-10-30 16:35:05.140730298 +0000 UTC m=+866.889319681" Oct 30 16:35:05.141020 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:05.140863 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:05.142634 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:05.142614 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:35:05.142688 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:05.142658 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:35:06.141141 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.141104 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:35:06.141583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.141160 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:35:06.269279 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.269254 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:06.352066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.351721 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-xks-ca\") pod \"a9ddd201-b315-49c5-83bf-6cacc3890c13\" (UID: \"a9ddd201-b315-49c5-83bf-6cacc3890c13\") " Oct 30 16:35:06.352066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.351797 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-openshift-ca\") pod \"a9ddd201-b315-49c5-83bf-6cacc3890c13\" (UID: \"a9ddd201-b315-49c5-83bf-6cacc3890c13\") " Oct 30 16:35:06.352066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.351916 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9ddd201-b315-49c5-83bf-6cacc3890c13-kube-api-access-gcw5d\" (UniqueName: \"kubernetes.io/projected/a9ddd201-b315-49c5-83bf-6cacc3890c13-kube-api-access-gcw5d\") pod \"a9ddd201-b315-49c5-83bf-6cacc3890c13\" (UID: \"a9ddd201-b315-49c5-83bf-6cacc3890c13\") " Oct 30 16:35:06.354552 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.354510 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-openshift-ca" pod "a9ddd201-b315-49c5-83bf-6cacc3890c13" (UID: "a9ddd201-b315-49c5-83bf-6cacc3890c13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:06.355563 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.355519 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ddd201-b315-49c5-83bf-6cacc3890c13-kube-api-access-gcw5d" pod "a9ddd201-b315-49c5-83bf-6cacc3890c13" (UID: "a9ddd201-b315-49c5-83bf-6cacc3890c13"). InnerVolumeSpecName "kube-api-access-gcw5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:06.357162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.357140 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-xks-ca" pod "a9ddd201-b315-49c5-83bf-6cacc3890c13" (UID: "a9ddd201-b315-49c5-83bf-6cacc3890c13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:06.453979 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.453941 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcw5d\" (UniqueName: \"kubernetes.io/projected/a9ddd201-b315-49c5-83bf-6cacc3890c13-kube-api-access-gcw5d\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:06.454135 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.454007 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:06.454135 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.454025 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9ddd201-b315-49c5-83bf-6cacc3890c13-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:06.597660 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.597621 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6"] Oct 30 16:35:06.598263 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.598242 5838 cpu_manager.go:414] "containerMap: removing container" podUID="f408729b-6480-4c89-8944-f9a7e1ab5736" containerName="revoke-keys" Oct 30 16:35:06.598263 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.598259 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f408729b-6480-4c89-8944-f9a7e1ab5736" containerName="revoke-keys" Oct 30 16:35:06.598453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.598268 5838 cpu_manager.go:414] "containerMap: removing container" podUID="a9ddd201-b315-49c5-83bf-6cacc3890c13" containerName="revoke-keys" Oct 30 16:35:06.598453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.598275 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9ddd201-b315-49c5-83bf-6cacc3890c13" containerName="revoke-keys" Oct 30 16:35:06.598453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.598383 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="f408729b-6480-4c89-8944-f9a7e1ab5736" containerName="revoke-keys" Oct 30 16:35:06.598453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.598391 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9ddd201-b315-49c5-83bf-6cacc3890c13" containerName="revoke-keys" Oct 30 16:35:06.601429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.601411 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:06.645657 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.644234 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6"] Oct 30 16:35:06.656823 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.656787 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31cb4ca-d6tn6\" (UID: \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:06.657033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.656945 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8666\" (UniqueName: \"kubernetes.io/projected/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-kube-api-access-g8666\") pod \"maas-api-revoke-sub-d31cb4ca-d6tn6\" (UID: \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:06.657115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.657099 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31cb4ca-d6tn6\" (UID: \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:06.758815 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.758760 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8666\" (UniqueName: \"kubernetes.io/projected/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-kube-api-access-g8666\") pod \"maas-api-revoke-sub-d31cb4ca-d6tn6\" (UID: \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:06.759049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.758842 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31cb4ca-d6tn6\" (UID: \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:06.759049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.758938 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31cb4ca-d6tn6\" (UID: \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:06.759589 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.759564 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31cb4ca-d6tn6\" (UID: \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:06.761326 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.761300 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31cb4ca-d6tn6\" (UID: \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:06.781965 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.781943 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8666\" (UniqueName: \"kubernetes.io/projected/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-kube-api-access-g8666\") pod \"maas-api-revoke-sub-d31cb4ca-d6tn6\" (UID: \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:06.936510 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:06.936163 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:07.087595 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:07.087468 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" Oct 30 16:35:07.087595 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:07.087512 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls" event={"ID":"a9ddd201-b315-49c5-83bf-6cacc3890c13","Type":"ContainerDied","Data":"c662466abd3ea25a84ce1ec2fce8efb83dea470a15d77273fca3e4e455378dd3"} Oct 30 16:35:07.087595 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:07.087573 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c662466abd3ea25a84ce1ec2fce8efb83dea470a15d77273fca3e4e455378dd3" Oct 30 16:35:07.141572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:07.141530 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:35:07.142107 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:07.141585 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:35:07.247022 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:07.246717 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6"] Oct 30 16:35:07.249950 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:35:07.249911 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc65f4c_eb79_43b2_b288_afae9f2a2fdf.slice/crio-3e9bf75ee04aa0f2c907dae5fc084392d1a3d7c6dbd2d45876a3f6a12bb47a7c WatchSource:0}: Error finding container 3e9bf75ee04aa0f2c907dae5fc084392d1a3d7c6dbd2d45876a3f6a12bb47a7c: Status 404 returned error can't find the container with id 3e9bf75ee04aa0f2c907dae5fc084392d1a3d7c6dbd2d45876a3f6a12bb47a7c Oct 30 16:35:08.096235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:08.096194 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" event={"ID":"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf","Type":"ContainerStarted","Data":"5f35db3f1cdf9a1d75fa6e035c35249d958afb4cc35808d53b5c4ba95e15571f"} Oct 30 16:35:08.096418 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:08.096278 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" event={"ID":"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf","Type":"ContainerStarted","Data":"3e9bf75ee04aa0f2c907dae5fc084392d1a3d7c6dbd2d45876a3f6a12bb47a7c"} Oct 30 16:35:08.141565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:08.141526 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:35:08.141748 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:08.141589 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:35:08.841432 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:08.841363 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" podStartSLOduration=2.841347279 podStartE2EDuration="2.841347279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:08.139265184 +0000 UTC m=+869.887854590" watchObservedRunningTime="2026-10-30 16:35:08.841347279 +0000 UTC m=+870.589936667" Oct 30 16:35:08.844113 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:08.844083 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:35:09.105122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:09.105025 5838 generic.go:356] "Generic (PLEG): container finished" podID="3cc65f4c-eb79-43b2-b288-afae9f2a2fdf" containerID="5f35db3f1cdf9a1d75fa6e035c35249d958afb4cc35808d53b5c4ba95e15571f" exitCode=0 Oct 30 16:35:09.105122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:09.105065 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" event={"ID":"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf","Type":"ContainerDied","Data":"5f35db3f1cdf9a1d75fa6e035c35249d958afb4cc35808d53b5c4ba95e15571f"} Oct 30 16:35:09.142067 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:09.142031 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:35:09.142545 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:09.142090 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:35:10.151369 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:10.151317 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:35:10.152007 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:10.151941 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:35:11.135581 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.135253 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:11.157465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.157429 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:35:11.159288 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.159253 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:35:11.164920 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.164891 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" event={"ID":"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf","Type":"ContainerDied","Data":"3e9bf75ee04aa0f2c907dae5fc084392d1a3d7c6dbd2d45876a3f6a12bb47a7c"} Oct 30 16:35:11.165230 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.165212 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e9bf75ee04aa0f2c907dae5fc084392d1a3d7c6dbd2d45876a3f6a12bb47a7c" Oct 30 16:35:11.165373 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.165151 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6" Oct 30 16:35:11.261994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.253316 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-openshift-ca\") pod \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\" (UID: \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\") " Oct 30 16:35:11.267289 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.267262 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-xks-ca\") pod \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\" (UID: \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\") " Oct 30 16:35:11.267839 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.267818 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-kube-api-access-g8666\" (UniqueName: \"kubernetes.io/projected/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-kube-api-access-g8666\") pod \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\" (UID: \"3cc65f4c-eb79-43b2-b288-afae9f2a2fdf\") " Oct 30 16:35:11.283045 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.267165 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-openshift-ca" pod "3cc65f4c-eb79-43b2-b288-afae9f2a2fdf" (UID: "3cc65f4c-eb79-43b2-b288-afae9f2a2fdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:11.304025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.303962 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-kube-api-access-g8666" pod "3cc65f4c-eb79-43b2-b288-afae9f2a2fdf" (UID: "3cc65f4c-eb79-43b2-b288-afae9f2a2fdf"). InnerVolumeSpecName "kube-api-access-g8666". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:11.306906 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.305190 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-xks-ca" pod "3cc65f4c-eb79-43b2-b288-afae9f2a2fdf" (UID: "3cc65f4c-eb79-43b2-b288-afae9f2a2fdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:11.374248 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.374200 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:11.374510 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.374489 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:11.374650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:11.374635 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8666\" (UniqueName: \"kubernetes.io/projected/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf-kube-api-access-g8666\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:12.134993 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:35:12.134242 5838 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc65f4c_eb79_43b2_b288_afae9f2a2fdf.slice\": RecentStats: unable to find data in memory cache]" Oct 30 16:35:13.148995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:13.148333 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:35:13.148995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:13.148415 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:35:13.741403 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:13.741368 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:13.741866 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:13.741790 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:13.743111 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:13.743088 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:35:15.409504 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.409468 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc"] Oct 30 16:35:15.410332 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.410304 5838 cpu_manager.go:414] "containerMap: removing container" podUID="3cc65f4c-eb79-43b2-b288-afae9f2a2fdf" containerName="revoke-keys" Oct 30 16:35:15.410332 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.410331 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc65f4c-eb79-43b2-b288-afae9f2a2fdf" containerName="revoke-keys" Oct 30 16:35:15.410562 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.410450 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc65f4c-eb79-43b2-b288-afae9f2a2fdf" containerName="revoke-keys" Oct 30 16:35:15.414285 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.414264 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:15.432785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.432755 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc"] Oct 30 16:35:15.440215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.440182 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtgl9\" (UniqueName: \"kubernetes.io/projected/305ae504-54ca-4dd1-9f5d-86fd928febf8-kube-api-access-rtgl9\") pod \"maas-api-revoke-sub-4dbc862c-4ksrc\" (UID: \"305ae504-54ca-4dd1-9f5d-86fd928febf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:15.440360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.440228 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:35:15.440360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.440298 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dbc862c-4ksrc\" (UID: \"305ae504-54ca-4dd1-9f5d-86fd928febf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:15.440477 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.440374 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:35:15.440538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.440517 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dbc862c-4ksrc\" (UID: \"305ae504-54ca-4dd1-9f5d-86fd928febf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:15.542050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.542012 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dbc862c-4ksrc\" (UID: \"305ae504-54ca-4dd1-9f5d-86fd928febf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:15.542215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.542086 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dbc862c-4ksrc\" (UID: \"305ae504-54ca-4dd1-9f5d-86fd928febf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:15.542215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.542180 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtgl9\" (UniqueName: \"kubernetes.io/projected/305ae504-54ca-4dd1-9f5d-86fd928febf8-kube-api-access-rtgl9\") pod \"maas-api-revoke-sub-4dbc862c-4ksrc\" (UID: \"305ae504-54ca-4dd1-9f5d-86fd928febf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:15.542829 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.542803 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dbc862c-4ksrc\" (UID: \"305ae504-54ca-4dd1-9f5d-86fd928febf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:15.544421 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.544396 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dbc862c-4ksrc\" (UID: \"305ae504-54ca-4dd1-9f5d-86fd928febf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:15.573484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.573455 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtgl9\" (UniqueName: \"kubernetes.io/projected/305ae504-54ca-4dd1-9f5d-86fd928febf8-kube-api-access-rtgl9\") pod \"maas-api-revoke-sub-4dbc862c-4ksrc\" (UID: \"305ae504-54ca-4dd1-9f5d-86fd928febf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:15.728541 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:15.728460 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:16.033115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:16.033084 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc"] Oct 30 16:35:16.039091 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:35:16.035314 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305ae504_54ca_4dd1_9f5d_86fd928febf8.slice/crio-566be62e3448ddfc9de90407629df19997eae621a67fb3f31834e461cf369ebb WatchSource:0}: Error finding container 566be62e3448ddfc9de90407629df19997eae621a67fb3f31834e461cf369ebb: Status 404 returned error can't find the container with id 566be62e3448ddfc9de90407629df19997eae621a67fb3f31834e461cf369ebb Oct 30 16:35:16.212432 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:16.212398 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" event={"ID":"305ae504-54ca-4dd1-9f5d-86fd928febf8","Type":"ContainerStarted","Data":"566be62e3448ddfc9de90407629df19997eae621a67fb3f31834e461cf369ebb"} Oct 30 16:35:17.220625 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:17.220536 5838 generic.go:356] "Generic (PLEG): container finished" podID="305ae504-54ca-4dd1-9f5d-86fd928febf8" containerID="72f7efe879e2e2e6554c3b0abff52f20ad3b2e578d829aaec00b998baf9b64e1" exitCode=0 Oct 30 16:35:17.221024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:17.220624 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" event={"ID":"305ae504-54ca-4dd1-9f5d-86fd928febf8","Type":"ContainerDied","Data":"72f7efe879e2e2e6554c3b0abff52f20ad3b2e578d829aaec00b998baf9b64e1"} Oct 30 16:35:18.380572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:18.380548 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:18.480308 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:18.480217 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305ae504-54ca-4dd1-9f5d-86fd928febf8-kube-api-access-rtgl9\" (UniqueName: \"kubernetes.io/projected/305ae504-54ca-4dd1-9f5d-86fd928febf8-kube-api-access-rtgl9\") pod \"305ae504-54ca-4dd1-9f5d-86fd928febf8\" (UID: \"305ae504-54ca-4dd1-9f5d-86fd928febf8\") " Oct 30 16:35:18.480308 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:18.480270 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-xks-ca\") pod \"305ae504-54ca-4dd1-9f5d-86fd928febf8\" (UID: \"305ae504-54ca-4dd1-9f5d-86fd928febf8\") " Oct 30 16:35:18.480555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:18.480406 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-openshift-ca\") pod \"305ae504-54ca-4dd1-9f5d-86fd928febf8\" (UID: \"305ae504-54ca-4dd1-9f5d-86fd928febf8\") " Oct 30 16:35:18.480752 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:18.480717 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-openshift-ca" pod "305ae504-54ca-4dd1-9f5d-86fd928febf8" (UID: "305ae504-54ca-4dd1-9f5d-86fd928febf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:18.482283 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:18.482258 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305ae504-54ca-4dd1-9f5d-86fd928febf8-kube-api-access-rtgl9" pod "305ae504-54ca-4dd1-9f5d-86fd928febf8" (UID: "305ae504-54ca-4dd1-9f5d-86fd928febf8"). InnerVolumeSpecName "kube-api-access-rtgl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:18.482403 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:18.482341 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-xks-ca" pod "305ae504-54ca-4dd1-9f5d-86fd928febf8" (UID: "305ae504-54ca-4dd1-9f5d-86fd928febf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:18.582190 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:18.582148 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtgl9\" (UniqueName: \"kubernetes.io/projected/305ae504-54ca-4dd1-9f5d-86fd928febf8-kube-api-access-rtgl9\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:18.582190 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:18.582185 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:18.582190 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:18.582199 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305ae504-54ca-4dd1-9f5d-86fd928febf8-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:35:19.235279 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:19.235246 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" Oct 30 16:35:19.235279 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:19.235259 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc" event={"ID":"305ae504-54ca-4dd1-9f5d-86fd928febf8","Type":"ContainerDied","Data":"566be62e3448ddfc9de90407629df19997eae621a67fb3f31834e461cf369ebb"} Oct 30 16:35:19.235592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:19.235308 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="566be62e3448ddfc9de90407629df19997eae621a67fb3f31834e461cf369ebb" Oct 30 16:35:31.797371 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:35:31.797331 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:35:38.510580 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:38.510549 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:35:38.513153 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:38.513128 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:35:38.638671 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:38.638634 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:35:48.515071 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.515030 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px"] Oct 30 16:35:48.515807 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.515788 5838 cpu_manager.go:414] "containerMap: removing container" podUID="305ae504-54ca-4dd1-9f5d-86fd928febf8" containerName="revoke-keys" Oct 30 16:35:48.515863 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.515808 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305ae504-54ca-4dd1-9f5d-86fd928febf8" containerName="revoke-keys" Oct 30 16:35:48.515953 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.515932 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="305ae504-54ca-4dd1-9f5d-86fd928febf8" containerName="revoke-keys" Oct 30 16:35:48.521809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.521779 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.546907 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.546875 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px"] Oct 30 16:35:48.667933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.667887 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fmd6\" (UniqueName: \"kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-kube-api-access-8fmd6\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.668184 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.668068 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-envoy\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.668184 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.668153 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-certs\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.668357 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.668199 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istiod-ca-cert\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.668357 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.668261 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.668472 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.668357 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-data\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.668472 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.668392 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-credential-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.668472 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.668439 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-podinfo\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.668622 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.668545 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-token\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.769319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.769228 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.769583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.769549 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-data\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.769583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.769584 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-credential-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.769738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.769611 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-podinfo\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.769738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.769636 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-token\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.769738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.769690 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fmd6\" (UniqueName: \"kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-kube-api-access-8fmd6\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.769886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.769735 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.769886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.769805 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-envoy\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.769886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.769856 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-certs\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.769886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.769883 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istiod-ca-cert\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.770120 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.769886 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-data\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.770120 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.770079 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-credential-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.770240 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.770138 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-certs\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.770511 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.770484 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istiod-ca-cert\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.772221 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.772175 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-envoy\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.772513 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.772490 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-podinfo\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.792764 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.792736 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-token\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.792892 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.792800 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fmd6\" (UniqueName: \"kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-kube-api-access-8fmd6\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:48.836269 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:48.836229 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:49.063246 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.063169 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn"] Oct 30 16:35:49.068304 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.068284 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.089941 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.089900 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn"] Oct 30 16:35:49.174692 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.174657 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-data\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.174866 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.174724 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-credential-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.174866 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.174807 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-certs\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.174866 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.174845 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.174984 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.174864 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istiod-ca-cert\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.175057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.175032 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-envoy\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.175089 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.175069 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x76l\" (UniqueName: \"kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-kube-api-access-2x76l\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.175140 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.175125 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-podinfo\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.175189 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.175177 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-token\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.206107 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.206082 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px"] Oct 30 16:35:49.209138 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:35:49.209104 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbf167c_2d5f_4cf5_a35f_d3d33ce303d1.slice/crio-0dcc7e369a5a9a393add35ff48b9cd5b841de72bfbde0c27f207a30e8488b5da WatchSource:0}: Error finding container 0dcc7e369a5a9a393add35ff48b9cd5b841de72bfbde0c27f207a30e8488b5da: Status 404 returned error can't find the container with id 0dcc7e369a5a9a393add35ff48b9cd5b841de72bfbde0c27f207a30e8488b5da Oct 30 16:35:49.211609 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.211578 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:49.211717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.211644 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:49.211717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.211674 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:49.276426 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.276393 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-envoy\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.276595 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.276437 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x76l\" (UniqueName: \"kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-kube-api-access-2x76l\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.276595 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.276484 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-podinfo\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.276713 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.276694 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-token\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.276842 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.276819 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-data\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.277006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.276897 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-credential-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.277006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.276957 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-certs\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.277143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.277044 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.277143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.277109 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istiod-ca-cert\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.277237 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.277161 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-data\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.277359 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.277335 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-credential-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.277458 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.277437 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.277514 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.277458 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-certs\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.277950 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.277891 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istiod-ca-cert\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.279271 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.279238 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-envoy\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.279436 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.279418 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-podinfo\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.302079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.302045 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x76l\" (UniqueName: \"kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-kube-api-access-2x76l\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.302263 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.302226 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-token\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.384133 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.384051 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:49.479685 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.479649 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" event={"ID":"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1","Type":"ContainerStarted","Data":"553b8b7b7e7299ffa15795a6ddf1011fc0fd5f06ffc3c92941eea6e6e316ffe6"} Oct 30 16:35:49.479882 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.479708 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" event={"ID":"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1","Type":"ContainerStarted","Data":"0dcc7e369a5a9a393add35ff48b9cd5b841de72bfbde0c27f207a30e8488b5da"} Oct 30 16:35:49.526680 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.526411 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" podStartSLOduration=1.526394829 podStartE2EDuration="1.526394829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:49.523711633 +0000 UTC m=+911.272301003" watchObservedRunningTime="2026-10-30 16:35:49.526394829 +0000 UTC m=+911.274984217" Oct 30 16:35:49.713639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.713611 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn"] Oct 30 16:35:49.716261 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:35:49.716236 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cafd8b_2615_423d_afd5_5a7fed76a7a6.slice/crio-c4903f18303d081e6dc50c2ac4a2c854850213f43b4c7b1266008455bcdcfaa0 WatchSource:0}: Error finding container c4903f18303d081e6dc50c2ac4a2c854850213f43b4c7b1266008455bcdcfaa0: Status 404 returned error can't find the container with id c4903f18303d081e6dc50c2ac4a2c854850213f43b4c7b1266008455bcdcfaa0 Oct 30 16:35:49.719106 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.719073 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:49.719182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.719141 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:49.719182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.719170 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:35:49.837212 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:49.837176 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:50.490914 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:50.490879 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" event={"ID":"b2cafd8b-2615-423d-afd5-5a7fed76a7a6","Type":"ContainerStarted","Data":"8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c"} Oct 30 16:35:50.491170 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:50.490933 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" event={"ID":"b2cafd8b-2615-423d-afd5-5a7fed76a7a6","Type":"ContainerStarted","Data":"c4903f18303d081e6dc50c2ac4a2c854850213f43b4c7b1266008455bcdcfaa0"} Oct 30 16:35:50.566186 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:50.566119 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" podStartSLOduration=1.566099672 podStartE2EDuration="1.566099672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:50.565413262 +0000 UTC m=+912.314002675" watchObservedRunningTime="2026-10-30 16:35:50.566099672 +0000 UTC m=+912.314689060" Oct 30 16:35:50.855553 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:50.855128 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:35:50.855553 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:50.855200 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" podUID="dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 16:35:51.385098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:51.385065 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:51.386630 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:51.386605 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:35:51.386820 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:51.386802 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" podUID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 16:35:51.836994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:51.836643 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:35:51.836994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:51.836704 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" podUID="dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 16:35:52.384815 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:52.384783 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:35:52.384966 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:52.384845 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" podUID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 16:35:52.840903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:52.840393 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:35:52.840903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:52.840471 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" podUID="dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 16:35:53.392997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:53.389400 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:35:53.392997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:53.389457 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" podUID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 16:35:53.840412 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:53.840370 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:35:53.840707 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:53.840677 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" podUID="dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 16:35:54.386424 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:54.386386 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:35:54.387052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:54.387016 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" podUID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 16:35:54.838557 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:54.838517 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:35:54.838847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:54.838805 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" podUID="dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 16:35:55.384547 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:55.384512 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:35:55.384731 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:55.384570 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" podUID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 16:35:55.839371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:55.839330 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:35:55.840013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:55.839962 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" podUID="dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 16:35:56.384429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:56.384386 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:35:56.384618 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:56.384458 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" podUID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 16:35:56.841876 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:56.841838 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:56.842368 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:56.842348 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:56.843123 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:56.843100 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:35:57.389628 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:57.389600 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:57.389889 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:57.389868 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:35:57.390639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:35:57.390618 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:36:01.569194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:01.569157 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:36:01.798048 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:36:01.798005 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:36:14.716349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.716310 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr"] Oct 30 16:36:14.720844 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.720821 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:14.739573 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.739543 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr"] Oct 30 16:36:14.746766 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.746741 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:14.746896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.746797 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:36:14.847150 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.847114 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4dfefcc-lgkmr\" (UID: \"987d59bf-7b46-45de-a4da-00d2e913b3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:14.847330 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.847172 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tltsg\" (UniqueName: \"kubernetes.io/projected/987d59bf-7b46-45de-a4da-00d2e913b3b9-kube-api-access-tltsg\") pod \"maas-api-revoke-sub-a4dfefcc-lgkmr\" (UID: \"987d59bf-7b46-45de-a4da-00d2e913b3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:14.847386 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.847332 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4dfefcc-lgkmr\" (UID: \"987d59bf-7b46-45de-a4da-00d2e913b3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:14.948453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.948405 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4dfefcc-lgkmr\" (UID: \"987d59bf-7b46-45de-a4da-00d2e913b3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:14.948629 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.948574 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4dfefcc-lgkmr\" (UID: \"987d59bf-7b46-45de-a4da-00d2e913b3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:14.948681 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.948640 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tltsg\" (UniqueName: \"kubernetes.io/projected/987d59bf-7b46-45de-a4da-00d2e913b3b9-kube-api-access-tltsg\") pod \"maas-api-revoke-sub-a4dfefcc-lgkmr\" (UID: \"987d59bf-7b46-45de-a4da-00d2e913b3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:14.949259 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.949227 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4dfefcc-lgkmr\" (UID: \"987d59bf-7b46-45de-a4da-00d2e913b3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:14.950804 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.950779 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4dfefcc-lgkmr\" (UID: \"987d59bf-7b46-45de-a4da-00d2e913b3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:14.971275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:14.971214 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tltsg\" (UniqueName: \"kubernetes.io/projected/987d59bf-7b46-45de-a4da-00d2e913b3b9-kube-api-access-tltsg\") pod \"maas-api-revoke-sub-a4dfefcc-lgkmr\" (UID: \"987d59bf-7b46-45de-a4da-00d2e913b3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:15.034527 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.034490 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:15.315805 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.315765 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn"] Oct 30 16:36:15.325702 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.325668 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.343206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.343174 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn"] Oct 30 16:36:15.344922 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.344890 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3e4f91\"/\"distinct-2-w2-3e4f91-kserve-self-signed-certs\"" Oct 30 16:36:15.352679 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.352647 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr"] Oct 30 16:36:15.354485 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:15.354446 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987d59bf_7b46_45de_a4da_00d2e913b3b9.slice/crio-4ef920f6b35319c5d6c6c8367a4470ea9828beb217ddced18e919ae3fda2f12a WatchSource:0}: Error finding container 4ef920f6b35319c5d6c6c8367a4470ea9828beb217ddced18e919ae3fda2f12a: Status 404 returned error can't find the container with id 4ef920f6b35319c5d6c6c8367a4470ea9828beb217ddced18e919ae3fda2f12a Oct 30 16:36:15.454593 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.454557 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-home\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.454593 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.454597 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79pw4\" (UniqueName: \"kubernetes.io/projected/c2511107-80b5-4a19-9907-880a96e9b5f2-kube-api-access-79pw4\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.454845 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.454802 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-model-cache\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.454905 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.454854 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-dshm\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.454956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.454923 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2511107-80b5-4a19-9907-880a96e9b5f2-tls-certs\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.455217 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.455199 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-tmp-dir\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.559819 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.559774 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2511107-80b5-4a19-9907-880a96e9b5f2-tls-certs\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.560202 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.560176 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-tmp-dir\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.560460 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.560440 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-home\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.560574 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.560559 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79pw4\" (UniqueName: \"kubernetes.io/projected/c2511107-80b5-4a19-9907-880a96e9b5f2-kube-api-access-79pw4\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.560795 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.560778 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-model-cache\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.560936 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.560921 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-dshm\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.562273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.561852 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-home\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.572101 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.565323 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-tmp-dir\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.572101 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.565371 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-model-cache\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.572101 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.566806 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2511107-80b5-4a19-9907-880a96e9b5f2-tls-certs\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.572803 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.572734 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-dshm\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.593995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.590288 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79pw4\" (UniqueName: \"kubernetes.io/projected/c2511107-80b5-4a19-9907-880a96e9b5f2-kube-api-access-79pw4\") pod \"distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.641920 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.641896 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:15.771887 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.771853 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" event={"ID":"987d59bf-7b46-45de-a4da-00d2e913b3b9","Type":"ContainerStarted","Data":"738034b92a1bc701fbb71e860be89f69f2469ab2515bc379bde68ea4af0ffc70"} Oct 30 16:36:15.772360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.771908 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" event={"ID":"987d59bf-7b46-45de-a4da-00d2e913b3b9","Type":"ContainerStarted","Data":"4ef920f6b35319c5d6c6c8367a4470ea9828beb217ddced18e919ae3fda2f12a"} Oct 30 16:36:15.811629 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:15.811567 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" podStartSLOduration=1.8115470299999998 podStartE2EDuration="1.81154703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:15.810595113 +0000 UTC m=+937.559184524" watchObservedRunningTime="2026-10-30 16:36:15.81154703 +0000 UTC m=+937.560136419" Oct 30 16:36:16.010338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:16.010310 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn"] Oct 30 16:36:16.012498 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:16.012464 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2511107_80b5_4a19_9907_880a96e9b5f2.slice/crio-70f6b9b249c6928e5315c9465a550d87e641b6b7d45bb5dbc8f8394a03b4465c WatchSource:0}: Error finding container 70f6b9b249c6928e5315c9465a550d87e641b6b7d45bb5dbc8f8394a03b4465c: Status 404 returned error can't find the container with id 70f6b9b249c6928e5315c9465a550d87e641b6b7d45bb5dbc8f8394a03b4465c Oct 30 16:36:16.781400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:16.781363 5838 generic.go:356] "Generic (PLEG): container finished" podID="987d59bf-7b46-45de-a4da-00d2e913b3b9" containerID="738034b92a1bc701fbb71e860be89f69f2469ab2515bc379bde68ea4af0ffc70" exitCode=0 Oct 30 16:36:16.781840 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:16.781430 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" event={"ID":"987d59bf-7b46-45de-a4da-00d2e913b3b9","Type":"ContainerDied","Data":"738034b92a1bc701fbb71e860be89f69f2469ab2515bc379bde68ea4af0ffc70"} Oct 30 16:36:16.783272 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:16.783249 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" event={"ID":"c2511107-80b5-4a19-9907-880a96e9b5f2","Type":"ContainerStarted","Data":"1f6eb8f9e2cee3bcb5b07e6163f231d74993b4858d7a7fc5da733d3f26e6032e"} Oct 30 16:36:16.783396 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:16.783278 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" event={"ID":"c2511107-80b5-4a19-9907-880a96e9b5f2","Type":"ContainerStarted","Data":"70f6b9b249c6928e5315c9465a550d87e641b6b7d45bb5dbc8f8394a03b4465c"} Oct 30 16:36:16.872538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:16.872476 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podStartSLOduration=1.87245631 podStartE2EDuration="1.87245631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:16.867155682 +0000 UTC m=+938.615745081" watchObservedRunningTime="2026-10-30 16:36:16.87245631 +0000 UTC m=+938.621045697" Oct 30 16:36:17.642577 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:17.642543 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:17.947934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:17.947907 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:18.013910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.013873 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b"] Oct 30 16:36:18.014854 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.014826 5838 cpu_manager.go:414] "containerMap: removing container" podUID="987d59bf-7b46-45de-a4da-00d2e913b3b9" containerName="revoke-keys" Oct 30 16:36:18.014854 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.014854 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987d59bf-7b46-45de-a4da-00d2e913b3b9" containerName="revoke-keys" Oct 30 16:36:18.015064 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.015009 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="987d59bf-7b46-45de-a4da-00d2e913b3b9" containerName="revoke-keys" Oct 30 16:36:18.018171 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.018144 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987d59bf-7b46-45de-a4da-00d2e913b3b9-kube-api-access-tltsg\" (UniqueName: \"kubernetes.io/projected/987d59bf-7b46-45de-a4da-00d2e913b3b9-kube-api-access-tltsg\") pod \"987d59bf-7b46-45de-a4da-00d2e913b3b9\" (UID: \"987d59bf-7b46-45de-a4da-00d2e913b3b9\") " Oct 30 16:36:18.018443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.018417 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-openshift-ca\") pod \"987d59bf-7b46-45de-a4da-00d2e913b3b9\" (UID: \"987d59bf-7b46-45de-a4da-00d2e913b3b9\") " Oct 30 16:36:18.018575 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.018471 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-xks-ca\") pod \"987d59bf-7b46-45de-a4da-00d2e913b3b9\" (UID: \"987d59bf-7b46-45de-a4da-00d2e913b3b9\") " Oct 30 16:36:18.019038 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.018795 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-openshift-ca" pod "987d59bf-7b46-45de-a4da-00d2e913b3b9" (UID: "987d59bf-7b46-45de-a4da-00d2e913b3b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:18.019038 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.018955 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:18.019483 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.019464 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:18.025150 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.025106 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-xks-ca" pod "987d59bf-7b46-45de-a4da-00d2e913b3b9" (UID: "987d59bf-7b46-45de-a4da-00d2e913b3b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:18.025150 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.025131 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987d59bf-7b46-45de-a4da-00d2e913b3b9-kube-api-access-tltsg" pod "987d59bf-7b46-45de-a4da-00d2e913b3b9" (UID: "987d59bf-7b46-45de-a4da-00d2e913b3b9"). InnerVolumeSpecName "kube-api-access-tltsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:18.040578 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.040545 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b"] Oct 30 16:36:18.136993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.133868 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e901e7a-z559b\" (UID: \"c3b65095-6bd0-449b-9051-4e77c282b043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:18.136993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.133920 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgj4\" (UniqueName: \"kubernetes.io/projected/c3b65095-6bd0-449b-9051-4e77c282b043-kube-api-access-ttgj4\") pod \"maas-api-revoke-sub-7e901e7a-z559b\" (UID: \"c3b65095-6bd0-449b-9051-4e77c282b043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:18.136993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.134025 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e901e7a-z559b\" (UID: \"c3b65095-6bd0-449b-9051-4e77c282b043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:18.136993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.134151 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tltsg\" (UniqueName: \"kubernetes.io/projected/987d59bf-7b46-45de-a4da-00d2e913b3b9-kube-api-access-tltsg\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:18.136993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.134167 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/987d59bf-7b46-45de-a4da-00d2e913b3b9-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:18.235425 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.235400 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e901e7a-z559b\" (UID: \"c3b65095-6bd0-449b-9051-4e77c282b043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:18.235580 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.235432 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgj4\" (UniqueName: \"kubernetes.io/projected/c3b65095-6bd0-449b-9051-4e77c282b043-kube-api-access-ttgj4\") pod \"maas-api-revoke-sub-7e901e7a-z559b\" (UID: \"c3b65095-6bd0-449b-9051-4e77c282b043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:18.235580 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.235482 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e901e7a-z559b\" (UID: \"c3b65095-6bd0-449b-9051-4e77c282b043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:18.236264 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.236240 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e901e7a-z559b\" (UID: \"c3b65095-6bd0-449b-9051-4e77c282b043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:18.237777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.237755 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e901e7a-z559b\" (UID: \"c3b65095-6bd0-449b-9051-4e77c282b043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:18.258521 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.258495 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgj4\" (UniqueName: \"kubernetes.io/projected/c3b65095-6bd0-449b-9051-4e77c282b043-kube-api-access-ttgj4\") pod \"maas-api-revoke-sub-7e901e7a-z559b\" (UID: \"c3b65095-6bd0-449b-9051-4e77c282b043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:18.355956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.355921 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:18.670343 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.670305 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b"] Oct 30 16:36:18.672471 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:18.672442 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b65095_6bd0_449b_9051_4e77c282b043.slice/crio-281be11a3d740892bca808d63c8d8859d5a636f4cb176e2d4d549e021d6a7080 WatchSource:0}: Error finding container 281be11a3d740892bca808d63c8d8859d5a636f4cb176e2d4d549e021d6a7080: Status 404 returned error can't find the container with id 281be11a3d740892bca808d63c8d8859d5a636f4cb176e2d4d549e021d6a7080 Oct 30 16:36:18.798858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.798809 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" event={"ID":"c3b65095-6bd0-449b-9051-4e77c282b043","Type":"ContainerStarted","Data":"281be11a3d740892bca808d63c8d8859d5a636f4cb176e2d4d549e021d6a7080"} Oct 30 16:36:18.800234 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.800202 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" event={"ID":"987d59bf-7b46-45de-a4da-00d2e913b3b9","Type":"ContainerDied","Data":"4ef920f6b35319c5d6c6c8367a4470ea9828beb217ddced18e919ae3fda2f12a"} Oct 30 16:36:18.800234 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.800224 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr" Oct 30 16:36:18.800441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.800256 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ef920f6b35319c5d6c6c8367a4470ea9828beb217ddced18e919ae3fda2f12a" Oct 30 16:36:18.881561 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.881529 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2"] Oct 30 16:36:18.885713 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.885695 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:18.909214 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.909179 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2"] Oct 30 16:36:18.944393 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.944253 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-podinfo\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:18.944393 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.944304 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-data\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:18.944644 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.944411 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-token\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:18.944644 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.944511 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:18.944644 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.944532 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-certs\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:18.944816 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.944699 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-credential-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:18.944816 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.944742 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8pd\" (UniqueName: \"kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-kube-api-access-qx8pd\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:18.944929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.944821 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-envoy\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:18.944929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:18.944882 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc48445-4345-4a22-bc03-6c2108da11b6-istiod-ca-cert\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.046354 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.046311 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.046354 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.046352 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-certs\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.046881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.046391 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-credential-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.046881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.046432 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8pd\" (UniqueName: \"kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-kube-api-access-qx8pd\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.046881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.046464 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-envoy\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.046881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.046504 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc48445-4345-4a22-bc03-6c2108da11b6-istiod-ca-cert\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.046881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.046562 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-podinfo\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.046881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.046677 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-data\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.046881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.046764 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-token\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.046881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.046810 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.046881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.046842 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-certs\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.047344 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.046892 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-credential-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.047344 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.047122 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-data\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.047344 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.047244 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc48445-4345-4a22-bc03-6c2108da11b6-istiod-ca-cert\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.049335 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.049290 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-envoy\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.049477 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.049384 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-podinfo\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.072947 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.072523 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8pd\" (UniqueName: \"kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-kube-api-access-qx8pd\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.075193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.075168 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-token\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.201328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.201045 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:19.530632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.530586 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2"] Oct 30 16:36:19.553841 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:19.553803 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc48445_4345_4a22_bc03_6c2108da11b6.slice/crio-2d6f33f4d154be08ce16c111a6ea497456d7d9f8fa1085ed4da2cd2bea2b9978 WatchSource:0}: Error finding container 2d6f33f4d154be08ce16c111a6ea497456d7d9f8fa1085ed4da2cd2bea2b9978: Status 404 returned error can't find the container with id 2d6f33f4d154be08ce16c111a6ea497456d7d9f8fa1085ed4da2cd2bea2b9978 Oct 30 16:36:19.556416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.556380 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:19.556523 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.556451 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:19.556523 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.556481 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:19.604006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.603953 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4"] Oct 30 16:36:19.608172 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.608151 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:19.629705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.629679 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4"] Oct 30 16:36:19.750486 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.750442 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb"] Oct 30 16:36:19.754733 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.754710 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:19.755679 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.755652 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fffa68cb-8lsq4\" (UID: \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:19.755810 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.755791 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhsg\" (UniqueName: \"kubernetes.io/projected/43bd21e5-a642-4ea3-a0ac-9f640ef94625-kube-api-access-qxhsg\") pod \"maas-api-revoke-sub-fffa68cb-8lsq4\" (UID: \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:19.756001 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.755980 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fffa68cb-8lsq4\" (UID: \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:19.776221 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.776187 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb"] Oct 30 16:36:19.809165 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.809076 5838 generic.go:356] "Generic (PLEG): container finished" podID="c3b65095-6bd0-449b-9051-4e77c282b043" containerID="0cfdc5a8b5a72b790fe6c7a1aa761aad1d3db5a2e9ba20d6968f2e61f95e66fc" exitCode=0 Oct 30 16:36:19.809311 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.809168 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" event={"ID":"c3b65095-6bd0-449b-9051-4e77c282b043","Type":"ContainerDied","Data":"0cfdc5a8b5a72b790fe6c7a1aa761aad1d3db5a2e9ba20d6968f2e61f95e66fc"} Oct 30 16:36:19.811038 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.811008 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" event={"ID":"9dc48445-4345-4a22-bc03-6c2108da11b6","Type":"ContainerStarted","Data":"b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05"} Oct 30 16:36:19.811187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.811061 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" event={"ID":"9dc48445-4345-4a22-bc03-6c2108da11b6","Type":"ContainerStarted","Data":"2d6f33f4d154be08ce16c111a6ea497456d7d9f8fa1085ed4da2cd2bea2b9978"} Oct 30 16:36:19.856791 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.856753 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fffa68cb-8lsq4\" (UID: \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:19.856991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.856802 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d38b37f4-k79wb\" (UID: \"abd3df30-43f9-46f3-9be2-b24b2b021395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:19.856991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.856831 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fffa68cb-8lsq4\" (UID: \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:19.856991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.856882 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d38b37f4-k79wb\" (UID: \"abd3df30-43f9-46f3-9be2-b24b2b021395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:19.857173 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.857047 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhsg\" (UniqueName: \"kubernetes.io/projected/43bd21e5-a642-4ea3-a0ac-9f640ef94625-kube-api-access-qxhsg\") pod \"maas-api-revoke-sub-fffa68cb-8lsq4\" (UID: \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:19.857232 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.857199 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkz5g\" (UniqueName: \"kubernetes.io/projected/abd3df30-43f9-46f3-9be2-b24b2b021395-kube-api-access-mkz5g\") pod \"maas-api-revoke-sub-d38b37f4-k79wb\" (UID: \"abd3df30-43f9-46f3-9be2-b24b2b021395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:19.857440 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.857415 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fffa68cb-8lsq4\" (UID: \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:19.859326 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.859307 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fffa68cb-8lsq4\" (UID: \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:19.880073 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.880036 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhsg\" (UniqueName: \"kubernetes.io/projected/43bd21e5-a642-4ea3-a0ac-9f640ef94625-kube-api-access-qxhsg\") pod \"maas-api-revoke-sub-fffa68cb-8lsq4\" (UID: \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:19.892298 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.892233 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" podStartSLOduration=1.8922173820000001 podStartE2EDuration="1.892217382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:19.891523742 +0000 UTC m=+941.640113129" watchObservedRunningTime="2026-10-30 16:36:19.892217382 +0000 UTC m=+941.640806769" Oct 30 16:36:19.921979 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.921929 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:19.973998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.970711 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d38b37f4-k79wb\" (UID: \"abd3df30-43f9-46f3-9be2-b24b2b021395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:19.985354 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.985284 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d38b37f4-k79wb\" (UID: \"abd3df30-43f9-46f3-9be2-b24b2b021395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:19.985527 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:19.985513 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d38b37f4-k79wb\" (UID: \"abd3df30-43f9-46f3-9be2-b24b2b021395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:20.023425 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:20.022322 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkz5g\" (UniqueName: \"kubernetes.io/projected/abd3df30-43f9-46f3-9be2-b24b2b021395-kube-api-access-mkz5g\") pod \"maas-api-revoke-sub-d38b37f4-k79wb\" (UID: \"abd3df30-43f9-46f3-9be2-b24b2b021395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:20.023941 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:20.023915 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d38b37f4-k79wb\" (UID: \"abd3df30-43f9-46f3-9be2-b24b2b021395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:20.047277 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:20.047241 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkz5g\" (UniqueName: \"kubernetes.io/projected/abd3df30-43f9-46f3-9be2-b24b2b021395-kube-api-access-mkz5g\") pod \"maas-api-revoke-sub-d38b37f4-k79wb\" (UID: \"abd3df30-43f9-46f3-9be2-b24b2b021395\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:20.071800 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:20.070072 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:20.202122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:20.202091 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:20.203607 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:20.203575 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:36:20.203744 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:20.203635 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" podUID="9dc48445-4345-4a22-bc03-6c2108da11b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:36:20.304195 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:20.304162 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4"] Oct 30 16:36:20.306074 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:20.306037 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bd21e5_a642_4ea3_a0ac_9f640ef94625.slice/crio-bd07840f9095b6628ad9ac4c2a876f6c753e4d086bc003579996c37214c9aeb2 WatchSource:0}: Error finding container bd07840f9095b6628ad9ac4c2a876f6c753e4d086bc003579996c37214c9aeb2: Status 404 returned error can't find the container with id bd07840f9095b6628ad9ac4c2a876f6c753e4d086bc003579996c37214c9aeb2 Oct 30 16:36:20.408799 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:20.408771 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb"] Oct 30 16:36:20.411086 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:20.411057 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd3df30_43f9_46f3_9be2_b24b2b021395.slice/crio-bdeb91d3390a10a6d61231eb3c9b6ff3a82c4d203997f73c876f3e61b3c404c3 WatchSource:0}: Error finding container bdeb91d3390a10a6d61231eb3c9b6ff3a82c4d203997f73c876f3e61b3c404c3: Status 404 returned error can't find the container with id bdeb91d3390a10a6d61231eb3c9b6ff3a82c4d203997f73c876f3e61b3c404c3 Oct 30 16:36:20.843113 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:20.843078 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" event={"ID":"abd3df30-43f9-46f3-9be2-b24b2b021395","Type":"ContainerStarted","Data":"bdeb91d3390a10a6d61231eb3c9b6ff3a82c4d203997f73c876f3e61b3c404c3"} Oct 30 16:36:20.844911 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:20.844882 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" event={"ID":"43bd21e5-a642-4ea3-a0ac-9f640ef94625","Type":"ContainerStarted","Data":"bd07840f9095b6628ad9ac4c2a876f6c753e4d086bc003579996c37214c9aeb2"} Oct 30 16:36:21.073191 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.073163 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:21.155886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.155838 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-xks-ca\") pod \"c3b65095-6bd0-449b-9051-4e77c282b043\" (UID: \"c3b65095-6bd0-449b-9051-4e77c282b043\") " Oct 30 16:36:21.156097 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.156009 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-openshift-ca\") pod \"c3b65095-6bd0-449b-9051-4e77c282b043\" (UID: \"c3b65095-6bd0-449b-9051-4e77c282b043\") " Oct 30 16:36:21.156097 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.156062 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b65095-6bd0-449b-9051-4e77c282b043-kube-api-access-ttgj4\" (UniqueName: \"kubernetes.io/projected/c3b65095-6bd0-449b-9051-4e77c282b043-kube-api-access-ttgj4\") pod \"c3b65095-6bd0-449b-9051-4e77c282b043\" (UID: \"c3b65095-6bd0-449b-9051-4e77c282b043\") " Oct 30 16:36:21.156429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.156390 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-openshift-ca" pod "c3b65095-6bd0-449b-9051-4e77c282b043" (UID: "c3b65095-6bd0-449b-9051-4e77c282b043"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:21.156580 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.156561 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:21.158305 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.158242 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-xks-ca" pod "c3b65095-6bd0-449b-9051-4e77c282b043" (UID: "c3b65095-6bd0-449b-9051-4e77c282b043"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:21.158410 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.158386 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b65095-6bd0-449b-9051-4e77c282b043-kube-api-access-ttgj4" pod "c3b65095-6bd0-449b-9051-4e77c282b043" (UID: "c3b65095-6bd0-449b-9051-4e77c282b043"). InnerVolumeSpecName "kube-api-access-ttgj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:21.202131 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.202089 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:36:21.202305 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.202149 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" podUID="9dc48445-4345-4a22-bc03-6c2108da11b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:36:21.257311 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.257276 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgj4\" (UniqueName: \"kubernetes.io/projected/c3b65095-6bd0-449b-9051-4e77c282b043-kube-api-access-ttgj4\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:21.257311 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.257307 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b65095-6bd0-449b-9051-4e77c282b043-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:21.856890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.855078 5838 generic.go:356] "Generic (PLEG): container finished" podID="43bd21e5-a642-4ea3-a0ac-9f640ef94625" containerID="fc099bcad56c75368b3cc5062ef324efe31283732aaa6aafd0d687aca7fa619b" exitCode=0 Oct 30 16:36:21.856890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.855162 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" event={"ID":"43bd21e5-a642-4ea3-a0ac-9f640ef94625","Type":"ContainerDied","Data":"fc099bcad56c75368b3cc5062ef324efe31283732aaa6aafd0d687aca7fa619b"} Oct 30 16:36:21.856890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.856831 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" event={"ID":"c3b65095-6bd0-449b-9051-4e77c282b043","Type":"ContainerDied","Data":"281be11a3d740892bca808d63c8d8859d5a636f4cb176e2d4d549e021d6a7080"} Oct 30 16:36:21.856890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.856867 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="281be11a3d740892bca808d63c8d8859d5a636f4cb176e2d4d549e021d6a7080" Oct 30 16:36:21.857132 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.856915 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b" Oct 30 16:36:21.862047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.860513 5838 generic.go:356] "Generic (PLEG): container finished" podID="abd3df30-43f9-46f3-9be2-b24b2b021395" containerID="5150e66809fbbdc9de891f7fe17d072a334f30efc0188a3f413fc94abe2733a4" exitCode=0 Oct 30 16:36:21.862047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:21.860588 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" event={"ID":"abd3df30-43f9-46f3-9be2-b24b2b021395","Type":"ContainerDied","Data":"5150e66809fbbdc9de891f7fe17d072a334f30efc0188a3f413fc94abe2733a4"} Oct 30 16:36:22.202198 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:22.202111 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:36:22.202198 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:22.202165 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" podUID="9dc48445-4345-4a22-bc03-6c2108da11b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:36:23.073371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.073334 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:23.108831 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.108799 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:23.182928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.182824 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-openshift-ca\") pod \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\" (UID: \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\") " Oct 30 16:36:23.182928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.182920 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-xks-ca\") pod \"abd3df30-43f9-46f3-9be2-b24b2b021395\" (UID: \"abd3df30-43f9-46f3-9be2-b24b2b021395\") " Oct 30 16:36:23.183195 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.183007 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-xks-ca\") pod \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\" (UID: \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\") " Oct 30 16:36:23.183195 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.183071 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bd21e5-a642-4ea3-a0ac-9f640ef94625-kube-api-access-qxhsg\" (UniqueName: \"kubernetes.io/projected/43bd21e5-a642-4ea3-a0ac-9f640ef94625-kube-api-access-qxhsg\") pod \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\" (UID: \"43bd21e5-a642-4ea3-a0ac-9f640ef94625\") " Oct 30 16:36:23.183304 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.183265 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-openshift-ca" pod "43bd21e5-a642-4ea3-a0ac-9f640ef94625" (UID: "43bd21e5-a642-4ea3-a0ac-9f640ef94625"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:23.183359 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.183325 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-openshift-ca\") pod \"abd3df30-43f9-46f3-9be2-b24b2b021395\" (UID: \"abd3df30-43f9-46f3-9be2-b24b2b021395\") " Oct 30 16:36:23.183409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.183396 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd3df30-43f9-46f3-9be2-b24b2b021395-kube-api-access-mkz5g\" (UniqueName: \"kubernetes.io/projected/abd3df30-43f9-46f3-9be2-b24b2b021395-kube-api-access-mkz5g\") pod \"abd3df30-43f9-46f3-9be2-b24b2b021395\" (UID: \"abd3df30-43f9-46f3-9be2-b24b2b021395\") " Oct 30 16:36:23.183765 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.183736 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-openshift-ca" pod "abd3df30-43f9-46f3-9be2-b24b2b021395" (UID: "abd3df30-43f9-46f3-9be2-b24b2b021395"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:23.183950 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.183931 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:23.184043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.183955 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:23.185808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.185763 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-xks-ca" pod "43bd21e5-a642-4ea3-a0ac-9f640ef94625" (UID: "43bd21e5-a642-4ea3-a0ac-9f640ef94625"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:23.186175 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.185856 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bd21e5-a642-4ea3-a0ac-9f640ef94625-kube-api-access-qxhsg" pod "43bd21e5-a642-4ea3-a0ac-9f640ef94625" (UID: "43bd21e5-a642-4ea3-a0ac-9f640ef94625"). InnerVolumeSpecName "kube-api-access-qxhsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:23.188577 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.188554 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-xks-ca" pod "abd3df30-43f9-46f3-9be2-b24b2b021395" (UID: "abd3df30-43f9-46f3-9be2-b24b2b021395"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:23.189040 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.188751 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd3df30-43f9-46f3-9be2-b24b2b021395-kube-api-access-mkz5g" pod "abd3df30-43f9-46f3-9be2-b24b2b021395" (UID: "abd3df30-43f9-46f3-9be2-b24b2b021395"). InnerVolumeSpecName "kube-api-access-mkz5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:23.201854 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.201499 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:36:23.201854 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.201557 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" podUID="9dc48445-4345-4a22-bc03-6c2108da11b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:36:23.285555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.285519 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd3df30-43f9-46f3-9be2-b24b2b021395-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:23.285555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.285549 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd21e5-a642-4ea3-a0ac-9f640ef94625-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:23.285555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.285558 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxhsg\" (UniqueName: \"kubernetes.io/projected/43bd21e5-a642-4ea3-a0ac-9f640ef94625-kube-api-access-qxhsg\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:23.286039 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.285567 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkz5g\" (UniqueName: \"kubernetes.io/projected/abd3df30-43f9-46f3-9be2-b24b2b021395-kube-api-access-mkz5g\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:23.886831 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.886776 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" event={"ID":"abd3df30-43f9-46f3-9be2-b24b2b021395","Type":"ContainerDied","Data":"bdeb91d3390a10a6d61231eb3c9b6ff3a82c4d203997f73c876f3e61b3c404c3"} Oct 30 16:36:23.887127 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.887105 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdeb91d3390a10a6d61231eb3c9b6ff3a82c4d203997f73c876f3e61b3c404c3" Oct 30 16:36:23.887304 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.887289 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb" Oct 30 16:36:23.902716 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.900740 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" event={"ID":"43bd21e5-a642-4ea3-a0ac-9f640ef94625","Type":"ContainerDied","Data":"bd07840f9095b6628ad9ac4c2a876f6c753e4d086bc003579996c37214c9aeb2"} Oct 30 16:36:23.902716 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.900809 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd07840f9095b6628ad9ac4c2a876f6c753e4d086bc003579996c37214c9aeb2" Oct 30 16:36:23.902716 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:23.900881 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4" Oct 30 16:36:24.201890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:24.201797 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:36:24.201890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:24.201868 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" podUID="9dc48445-4345-4a22-bc03-6c2108da11b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:36:25.201767 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:25.201736 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:36:25.202146 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:25.201788 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" podUID="9dc48445-4345-4a22-bc03-6c2108da11b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:36:25.642797 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:25.642759 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:25.656497 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:25.656465 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:25.928371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:25.928283 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:36:26.202103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:26.202016 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:36:26.202103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:26.202072 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" podUID="9dc48445-4345-4a22-bc03-6c2108da11b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:36:26.678607 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:26.678565 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:36:27.206233 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:27.206201 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:27.206682 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:27.206657 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:27.207565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:27.207539 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:36:28.399274 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.399239 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl"] Oct 30 16:36:28.400198 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.400177 5838 cpu_manager.go:414] "containerMap: removing container" podUID="c3b65095-6bd0-449b-9051-4e77c282b043" containerName="revoke-keys" Oct 30 16:36:28.400247 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.400202 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b65095-6bd0-449b-9051-4e77c282b043" containerName="revoke-keys" Oct 30 16:36:28.400247 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.400217 5838 cpu_manager.go:414] "containerMap: removing container" podUID="43bd21e5-a642-4ea3-a0ac-9f640ef94625" containerName="revoke-keys" Oct 30 16:36:28.400247 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.400226 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bd21e5-a642-4ea3-a0ac-9f640ef94625" containerName="revoke-keys" Oct 30 16:36:28.400340 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.400285 5838 cpu_manager.go:414] "containerMap: removing container" podUID="abd3df30-43f9-46f3-9be2-b24b2b021395" containerName="revoke-keys" Oct 30 16:36:28.400340 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.400294 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd3df30-43f9-46f3-9be2-b24b2b021395" containerName="revoke-keys" Oct 30 16:36:28.400495 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.400484 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b65095-6bd0-449b-9051-4e77c282b043" containerName="revoke-keys" Oct 30 16:36:28.400531 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.400497 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bd21e5-a642-4ea3-a0ac-9f640ef94625" containerName="revoke-keys" Oct 30 16:36:28.400531 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.400513 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd3df30-43f9-46f3-9be2-b24b2b021395" containerName="revoke-keys" Oct 30 16:36:28.404913 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.404886 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:28.426837 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.426800 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl"] Oct 30 16:36:28.427200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.427168 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:28.427502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.427484 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:36:28.537748 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.537702 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5eb544e-56hzl\" (UID: \"aeaf9129-0999-46d3-801c-1bb480135c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:28.537933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.537752 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5eb544e-56hzl\" (UID: \"aeaf9129-0999-46d3-801c-1bb480135c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:28.538061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.537930 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lnn7\" (UniqueName: \"kubernetes.io/projected/aeaf9129-0999-46d3-801c-1bb480135c6f-kube-api-access-9lnn7\") pod \"maas-api-revoke-sub-a5eb544e-56hzl\" (UID: \"aeaf9129-0999-46d3-801c-1bb480135c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:28.639357 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.639291 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5eb544e-56hzl\" (UID: \"aeaf9129-0999-46d3-801c-1bb480135c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:28.639542 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.639367 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5eb544e-56hzl\" (UID: \"aeaf9129-0999-46d3-801c-1bb480135c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:28.639542 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.639457 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lnn7\" (UniqueName: \"kubernetes.io/projected/aeaf9129-0999-46d3-801c-1bb480135c6f-kube-api-access-9lnn7\") pod \"maas-api-revoke-sub-a5eb544e-56hzl\" (UID: \"aeaf9129-0999-46d3-801c-1bb480135c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:28.640301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.640271 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5eb544e-56hzl\" (UID: \"aeaf9129-0999-46d3-801c-1bb480135c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:28.642247 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.642004 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5eb544e-56hzl\" (UID: \"aeaf9129-0999-46d3-801c-1bb480135c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:28.662912 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.662833 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lnn7\" (UniqueName: \"kubernetes.io/projected/aeaf9129-0999-46d3-801c-1bb480135c6f-kube-api-access-9lnn7\") pod \"maas-api-revoke-sub-a5eb544e-56hzl\" (UID: \"aeaf9129-0999-46d3-801c-1bb480135c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:28.737582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:28.737538 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:29.062202 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:29.062168 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl"] Oct 30 16:36:29.064418 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:29.064388 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeaf9129_0999_46d3_801c_1bb480135c6f.slice/crio-d73570d15620d2353e8023349ee401f0dfbba96a0a82b20add66af37dc869d86 WatchSource:0}: Error finding container d73570d15620d2353e8023349ee401f0dfbba96a0a82b20add66af37dc869d86: Status 404 returned error can't find the container with id d73570d15620d2353e8023349ee401f0dfbba96a0a82b20add66af37dc869d86 Oct 30 16:36:29.958728 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:29.958695 5838 generic.go:356] "Generic (PLEG): container finished" podID="aeaf9129-0999-46d3-801c-1bb480135c6f" containerID="6fb8c29dfdb12c47ce69dc1a9c5af09ee8aefed892857da5f719e2f471b9ada4" exitCode=0 Oct 30 16:36:29.959129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:29.958787 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" event={"ID":"aeaf9129-0999-46d3-801c-1bb480135c6f","Type":"ContainerDied","Data":"6fb8c29dfdb12c47ce69dc1a9c5af09ee8aefed892857da5f719e2f471b9ada4"} Oct 30 16:36:29.959129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:29.958837 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" event={"ID":"aeaf9129-0999-46d3-801c-1bb480135c6f","Type":"ContainerStarted","Data":"d73570d15620d2353e8023349ee401f0dfbba96a0a82b20add66af37dc869d86"} Oct 30 16:36:31.119409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.119381 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:31.176311 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.176269 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeaf9129-0999-46d3-801c-1bb480135c6f-kube-api-access-9lnn7\" (UniqueName: \"kubernetes.io/projected/aeaf9129-0999-46d3-801c-1bb480135c6f-kube-api-access-9lnn7\") pod \"aeaf9129-0999-46d3-801c-1bb480135c6f\" (UID: \"aeaf9129-0999-46d3-801c-1bb480135c6f\") " Oct 30 16:36:31.179070 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.179034 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeaf9129-0999-46d3-801c-1bb480135c6f-kube-api-access-9lnn7" pod "aeaf9129-0999-46d3-801c-1bb480135c6f" (UID: "aeaf9129-0999-46d3-801c-1bb480135c6f"). InnerVolumeSpecName "kube-api-access-9lnn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:31.277475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.277427 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-xks-ca\") pod \"aeaf9129-0999-46d3-801c-1bb480135c6f\" (UID: \"aeaf9129-0999-46d3-801c-1bb480135c6f\") " Oct 30 16:36:31.277681 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.277526 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-openshift-ca\") pod \"aeaf9129-0999-46d3-801c-1bb480135c6f\" (UID: \"aeaf9129-0999-46d3-801c-1bb480135c6f\") " Oct 30 16:36:31.277946 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.277922 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-openshift-ca" pod "aeaf9129-0999-46d3-801c-1bb480135c6f" (UID: "aeaf9129-0999-46d3-801c-1bb480135c6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:31.278060 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.278044 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lnn7\" (UniqueName: \"kubernetes.io/projected/aeaf9129-0999-46d3-801c-1bb480135c6f-kube-api-access-9lnn7\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:31.278110 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.278062 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:31.279719 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.279678 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-xks-ca" pod "aeaf9129-0999-46d3-801c-1bb480135c6f" (UID: "aeaf9129-0999-46d3-801c-1bb480135c6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:31.382143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.382106 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeaf9129-0999-46d3-801c-1bb480135c6f-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:31.798150 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:36:31.798108 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:36:31.975468 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.975437 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" Oct 30 16:36:31.975637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.975468 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl" event={"ID":"aeaf9129-0999-46d3-801c-1bb480135c6f","Type":"ContainerDied","Data":"d73570d15620d2353e8023349ee401f0dfbba96a0a82b20add66af37dc869d86"} Oct 30 16:36:31.975637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:31.975517 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d73570d15620d2353e8023349ee401f0dfbba96a0a82b20add66af37dc869d86" Oct 30 16:36:32.113761 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.113676 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5"] Oct 30 16:36:32.114393 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.114372 5838 cpu_manager.go:414] "containerMap: removing container" podUID="aeaf9129-0999-46d3-801c-1bb480135c6f" containerName="revoke-keys" Oct 30 16:36:32.114393 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.114393 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeaf9129-0999-46d3-801c-1bb480135c6f" containerName="revoke-keys" Oct 30 16:36:32.114556 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.114528 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeaf9129-0999-46d3-801c-1bb480135c6f" containerName="revoke-keys" Oct 30 16:36:32.118386 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.118364 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:32.132625 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.132581 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:32.133066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.132762 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:36:32.137495 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.137458 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5"] Oct 30 16:36:32.190842 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.190808 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6624684f-t8px5\" (UID: \"48a8b00a-4d8e-4428-80eb-b651e333a38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:32.191016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.190933 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g447s\" (UniqueName: \"kubernetes.io/projected/48a8b00a-4d8e-4428-80eb-b651e333a38f-kube-api-access-g447s\") pod \"maas-api-revoke-sub-6624684f-t8px5\" (UID: \"48a8b00a-4d8e-4428-80eb-b651e333a38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:32.191016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.191012 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6624684f-t8px5\" (UID: \"48a8b00a-4d8e-4428-80eb-b651e333a38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:32.292268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.292234 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g447s\" (UniqueName: \"kubernetes.io/projected/48a8b00a-4d8e-4428-80eb-b651e333a38f-kube-api-access-g447s\") pod \"maas-api-revoke-sub-6624684f-t8px5\" (UID: \"48a8b00a-4d8e-4428-80eb-b651e333a38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:32.292453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.292277 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6624684f-t8px5\" (UID: \"48a8b00a-4d8e-4428-80eb-b651e333a38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:32.292453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.292432 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6624684f-t8px5\" (UID: \"48a8b00a-4d8e-4428-80eb-b651e333a38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:32.293122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.293095 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6624684f-t8px5\" (UID: \"48a8b00a-4d8e-4428-80eb-b651e333a38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:32.294775 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.294750 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6624684f-t8px5\" (UID: \"48a8b00a-4d8e-4428-80eb-b651e333a38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:32.314841 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.314811 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g447s\" (UniqueName: \"kubernetes.io/projected/48a8b00a-4d8e-4428-80eb-b651e333a38f-kube-api-access-g447s\") pod \"maas-api-revoke-sub-6624684f-t8px5\" (UID: \"48a8b00a-4d8e-4428-80eb-b651e333a38f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:32.432543 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.432450 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:32.569380 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.569346 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9"] Oct 30 16:36:32.574749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.574722 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:32.594622 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.594586 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9"] Oct 30 16:36:32.614435 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.614392 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkrsz\" (UniqueName: \"kubernetes.io/projected/19e2aba0-8b73-4995-950e-6408c1e6f10f-kube-api-access-qkrsz\") pod \"maas-api-revoke-sub-d117a55f-kppf9\" (UID: \"19e2aba0-8b73-4995-950e-6408c1e6f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:32.614937 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.614494 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d117a55f-kppf9\" (UID: \"19e2aba0-8b73-4995-950e-6408c1e6f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:32.614937 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.614604 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d117a55f-kppf9\" (UID: \"19e2aba0-8b73-4995-950e-6408c1e6f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:32.742041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.717542 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkrsz\" (UniqueName: \"kubernetes.io/projected/19e2aba0-8b73-4995-950e-6408c1e6f10f-kube-api-access-qkrsz\") pod \"maas-api-revoke-sub-d117a55f-kppf9\" (UID: \"19e2aba0-8b73-4995-950e-6408c1e6f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:32.742041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.736339 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d117a55f-kppf9\" (UID: \"19e2aba0-8b73-4995-950e-6408c1e6f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:32.742041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.736399 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d117a55f-kppf9\" (UID: \"19e2aba0-8b73-4995-950e-6408c1e6f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:32.742041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.737366 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d117a55f-kppf9\" (UID: \"19e2aba0-8b73-4995-950e-6408c1e6f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:32.779545 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.777989 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkrsz\" (UniqueName: \"kubernetes.io/projected/19e2aba0-8b73-4995-950e-6408c1e6f10f-kube-api-access-qkrsz\") pod \"maas-api-revoke-sub-d117a55f-kppf9\" (UID: \"19e2aba0-8b73-4995-950e-6408c1e6f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:32.779545 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.779435 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d117a55f-kppf9\" (UID: \"19e2aba0-8b73-4995-950e-6408c1e6f10f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:32.825019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.824405 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5"] Oct 30 16:36:32.831296 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:32.831255 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a8b00a_4d8e_4428_80eb_b651e333a38f.slice/crio-b466c91b5fa56f3004ccf8699f1f98396733aa42b013cb7676968a7a8c7ab53a WatchSource:0}: Error finding container b466c91b5fa56f3004ccf8699f1f98396733aa42b013cb7676968a7a8c7ab53a: Status 404 returned error can't find the container with id b466c91b5fa56f3004ccf8699f1f98396733aa42b013cb7676968a7a8c7ab53a Oct 30 16:36:32.889198 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.889166 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:32.985036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:32.985003 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" event={"ID":"48a8b00a-4d8e-4428-80eb-b651e333a38f","Type":"ContainerStarted","Data":"b466c91b5fa56f3004ccf8699f1f98396733aa42b013cb7676968a7a8c7ab53a"} Oct 30 16:36:33.292061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:33.292030 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9"] Oct 30 16:36:33.293915 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:33.293885 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e2aba0_8b73_4995_950e_6408c1e6f10f.slice/crio-dcd93df9367d5489ad8b57a6ba0285d6056960c9a79bb5b12705ffd2cdb83de4 WatchSource:0}: Error finding container dcd93df9367d5489ad8b57a6ba0285d6056960c9a79bb5b12705ffd2cdb83de4: Status 404 returned error can't find the container with id dcd93df9367d5489ad8b57a6ba0285d6056960c9a79bb5b12705ffd2cdb83de4 Oct 30 16:36:33.995003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:33.994930 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" event={"ID":"19e2aba0-8b73-4995-950e-6408c1e6f10f","Type":"ContainerStarted","Data":"f07ecf60957a8a5bc7b723f2b33700dc1202e74db1e5512b314bd53b17436b53"} Oct 30 16:36:33.995003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:33.995006 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" event={"ID":"19e2aba0-8b73-4995-950e-6408c1e6f10f","Type":"ContainerStarted","Data":"dcd93df9367d5489ad8b57a6ba0285d6056960c9a79bb5b12705ffd2cdb83de4"} Oct 30 16:36:33.996640 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:33.996613 5838 generic.go:356] "Generic (PLEG): container finished" podID="48a8b00a-4d8e-4428-80eb-b651e333a38f" containerID="ec5f874975d96ff2692b0c1f102f1bf8f5bace786b377befac3e0066f79a81b6" exitCode=0 Oct 30 16:36:33.996765 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:33.996693 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" event={"ID":"48a8b00a-4d8e-4428-80eb-b651e333a38f","Type":"ContainerDied","Data":"ec5f874975d96ff2692b0c1f102f1bf8f5bace786b377befac3e0066f79a81b6"} Oct 30 16:36:34.037958 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.037886 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" podStartSLOduration=2.03786599 podStartE2EDuration="2.03786599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:34.035407321 +0000 UTC m=+955.783996709" watchObservedRunningTime="2026-10-30 16:36:34.03786599 +0000 UTC m=+955.786455377" Oct 30 16:36:34.066804 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.066771 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr"] Oct 30 16:36:34.071285 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.071266 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.094728 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.094698 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr"] Oct 30 16:36:34.160592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.160553 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-podinfo\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.160592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.160595 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d212d906-4b33-44b1-9079-1b5ae0000f3b-istiod-ca-cert\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.160814 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.160617 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-certs\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.160814 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.160764 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.160814 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.160802 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcg7r\" (UniqueName: \"kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-kube-api-access-lcg7r\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.160911 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.160820 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-token\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.160953 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.160933 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-envoy\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.161014 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.160995 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-data\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.161148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.161132 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-credential-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.178469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.178424 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f"] Oct 30 16:36:34.182676 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.182657 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:34.201724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.201690 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f"] Oct 30 16:36:34.262193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262158 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69c2159f-sht4f\" (UID: \"995c7339-905b-459a-88bf-9971b42ca95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:34.262348 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262291 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-credential-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.262389 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262347 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69c2159f-sht4f\" (UID: \"995c7339-905b-459a-88bf-9971b42ca95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:34.262389 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262377 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-podinfo\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.262462 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262395 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d212d906-4b33-44b1-9079-1b5ae0000f3b-istiod-ca-cert\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.262462 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262412 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-certs\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.262548 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262512 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.262616 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262579 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcg7r\" (UniqueName: \"kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-kube-api-access-lcg7r\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.262668 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262627 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-token\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.262723 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262667 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m8p6\" (UniqueName: \"kubernetes.io/projected/995c7339-905b-459a-88bf-9971b42ca95b-kube-api-access-4m8p6\") pod \"maas-api-revoke-sub-69c2159f-sht4f\" (UID: \"995c7339-905b-459a-88bf-9971b42ca95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:34.262723 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262705 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-envoy\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.262826 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262762 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-data\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.262826 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262738 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-credential-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.262826 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262801 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.262994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.262933 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-certs\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.263186 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.263165 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-data\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.263301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.263278 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d212d906-4b33-44b1-9079-1b5ae0000f3b-istiod-ca-cert\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.265174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.265149 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-envoy\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.265326 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.265308 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-podinfo\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.287791 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.286262 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcg7r\" (UniqueName: \"kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-kube-api-access-lcg7r\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.287791 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.286833 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-token\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.364275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.364231 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m8p6\" (UniqueName: \"kubernetes.io/projected/995c7339-905b-459a-88bf-9971b42ca95b-kube-api-access-4m8p6\") pod \"maas-api-revoke-sub-69c2159f-sht4f\" (UID: \"995c7339-905b-459a-88bf-9971b42ca95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:34.364713 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.364347 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69c2159f-sht4f\" (UID: \"995c7339-905b-459a-88bf-9971b42ca95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:34.364713 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.364474 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69c2159f-sht4f\" (UID: \"995c7339-905b-459a-88bf-9971b42ca95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:34.365073 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.365052 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69c2159f-sht4f\" (UID: \"995c7339-905b-459a-88bf-9971b42ca95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:34.366948 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.366920 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69c2159f-sht4f\" (UID: \"995c7339-905b-459a-88bf-9971b42ca95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:34.386148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.386115 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:34.386943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.386920 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m8p6\" (UniqueName: \"kubernetes.io/projected/995c7339-905b-459a-88bf-9971b42ca95b-kube-api-access-4m8p6\") pod \"maas-api-revoke-sub-69c2159f-sht4f\" (UID: \"995c7339-905b-459a-88bf-9971b42ca95b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:34.496947 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.496910 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:34.722192 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.722166 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr"] Oct 30 16:36:34.723909 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:34.723883 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd212d906_4b33_44b1_9079_1b5ae0000f3b.slice/crio-127ad2f939f5b3de988e220e367d44f96343350a04ce978db9e184e92d52979c WatchSource:0}: Error finding container 127ad2f939f5b3de988e220e367d44f96343350a04ce978db9e184e92d52979c: Status 404 returned error can't find the container with id 127ad2f939f5b3de988e220e367d44f96343350a04ce978db9e184e92d52979c Oct 30 16:36:34.726464 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.726428 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:34.726575 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.726507 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:34.726575 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.726553 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:36:34.807329 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:34.807303 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f"] Oct 30 16:36:34.809124 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:34.809098 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995c7339_905b_459a_88bf_9971b42ca95b.slice/crio-6b8c461ea1d8503179af9cba2a7fc7baaba8f034a53bc7edfba32aca22f735c6 WatchSource:0}: Error finding container 6b8c461ea1d8503179af9cba2a7fc7baaba8f034a53bc7edfba32aca22f735c6: Status 404 returned error can't find the container with id 6b8c461ea1d8503179af9cba2a7fc7baaba8f034a53bc7edfba32aca22f735c6 Oct 30 16:36:35.005895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.005854 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" event={"ID":"d212d906-4b33-44b1-9079-1b5ae0000f3b","Type":"ContainerStarted","Data":"aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96"} Oct 30 16:36:35.006093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.005927 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" event={"ID":"d212d906-4b33-44b1-9079-1b5ae0000f3b","Type":"ContainerStarted","Data":"127ad2f939f5b3de988e220e367d44f96343350a04ce978db9e184e92d52979c"} Oct 30 16:36:35.007332 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.007307 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" event={"ID":"995c7339-905b-459a-88bf-9971b42ca95b","Type":"ContainerStarted","Data":"6b8c461ea1d8503179af9cba2a7fc7baaba8f034a53bc7edfba32aca22f735c6"} Oct 30 16:36:35.009017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.008991 5838 generic.go:356] "Generic (PLEG): container finished" podID="19e2aba0-8b73-4995-950e-6408c1e6f10f" containerID="f07ecf60957a8a5bc7b723f2b33700dc1202e74db1e5512b314bd53b17436b53" exitCode=0 Oct 30 16:36:35.009169 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.009138 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" event={"ID":"19e2aba0-8b73-4995-950e-6408c1e6f10f","Type":"ContainerDied","Data":"f07ecf60957a8a5bc7b723f2b33700dc1202e74db1e5512b314bd53b17436b53"} Oct 30 16:36:35.055208 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.055091 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" podStartSLOduration=1.055070453 podStartE2EDuration="1.055070453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:35.050592348 +0000 UTC m=+956.799181764" watchObservedRunningTime="2026-10-30 16:36:35.055070453 +0000 UTC m=+956.803659846" Oct 30 16:36:35.170179 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.170142 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:35.283799 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.283754 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-openshift-ca\") pod \"48a8b00a-4d8e-4428-80eb-b651e333a38f\" (UID: \"48a8b00a-4d8e-4428-80eb-b651e333a38f\") " Oct 30 16:36:35.284010 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.283862 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a8b00a-4d8e-4428-80eb-b651e333a38f-kube-api-access-g447s\" (UniqueName: \"kubernetes.io/projected/48a8b00a-4d8e-4428-80eb-b651e333a38f-kube-api-access-g447s\") pod \"48a8b00a-4d8e-4428-80eb-b651e333a38f\" (UID: \"48a8b00a-4d8e-4428-80eb-b651e333a38f\") " Oct 30 16:36:35.284010 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.283944 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-xks-ca\") pod \"48a8b00a-4d8e-4428-80eb-b651e333a38f\" (UID: \"48a8b00a-4d8e-4428-80eb-b651e333a38f\") " Oct 30 16:36:35.284241 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.284213 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-openshift-ca" pod "48a8b00a-4d8e-4428-80eb-b651e333a38f" (UID: "48a8b00a-4d8e-4428-80eb-b651e333a38f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:35.284490 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.284469 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:35.286183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.286145 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-xks-ca" pod "48a8b00a-4d8e-4428-80eb-b651e333a38f" (UID: "48a8b00a-4d8e-4428-80eb-b651e333a38f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:35.286284 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.286213 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a8b00a-4d8e-4428-80eb-b651e333a38f-kube-api-access-g447s" pod "48a8b00a-4d8e-4428-80eb-b651e333a38f" (UID: "48a8b00a-4d8e-4428-80eb-b651e333a38f"). InnerVolumeSpecName "kube-api-access-g447s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:35.386183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.386145 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g447s\" (UniqueName: \"kubernetes.io/projected/48a8b00a-4d8e-4428-80eb-b651e333a38f-kube-api-access-g447s\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:35.386183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.386176 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a8b00a-4d8e-4428-80eb-b651e333a38f-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:35.386580 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.386305 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:35.388227 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.388199 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:36:35.388321 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:35.388253 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" podUID="d212d906-4b33-44b1-9079-1b5ae0000f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:36:36.044360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.044325 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" Oct 30 16:36:36.044360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.044338 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5" event={"ID":"48a8b00a-4d8e-4428-80eb-b651e333a38f","Type":"ContainerDied","Data":"b466c91b5fa56f3004ccf8699f1f98396733aa42b013cb7676968a7a8c7ab53a"} Oct 30 16:36:36.044566 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.044389 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b466c91b5fa56f3004ccf8699f1f98396733aa42b013cb7676968a7a8c7ab53a" Oct 30 16:36:36.046499 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.046464 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" event={"ID":"995c7339-905b-459a-88bf-9971b42ca95b","Type":"ContainerStarted","Data":"4c6cafa548e2a41bddffa3284b7c1704dcc1d4a5220e7d9736f2962137a10f7a"} Oct 30 16:36:36.088024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.087942 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" podStartSLOduration=2.087923233 podStartE2EDuration="2.087923233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:36.085552754 +0000 UTC m=+957.834142167" watchObservedRunningTime="2026-10-30 16:36:36.087923233 +0000 UTC m=+957.836512617" Oct 30 16:36:36.253717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.253688 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:36.299033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.298918 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-xks-ca\") pod \"19e2aba0-8b73-4995-950e-6408c1e6f10f\" (UID: \"19e2aba0-8b73-4995-950e-6408c1e6f10f\") " Oct 30 16:36:36.299204 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.299151 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-openshift-ca\") pod \"19e2aba0-8b73-4995-950e-6408c1e6f10f\" (UID: \"19e2aba0-8b73-4995-950e-6408c1e6f10f\") " Oct 30 16:36:36.299287 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.299269 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e2aba0-8b73-4995-950e-6408c1e6f10f-kube-api-access-qkrsz\" (UniqueName: \"kubernetes.io/projected/19e2aba0-8b73-4995-950e-6408c1e6f10f-kube-api-access-qkrsz\") pod \"19e2aba0-8b73-4995-950e-6408c1e6f10f\" (UID: \"19e2aba0-8b73-4995-950e-6408c1e6f10f\") " Oct 30 16:36:36.299704 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.299605 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-openshift-ca" pod "19e2aba0-8b73-4995-950e-6408c1e6f10f" (UID: "19e2aba0-8b73-4995-950e-6408c1e6f10f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:36.299873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.299853 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:36.301275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.301251 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-xks-ca" pod "19e2aba0-8b73-4995-950e-6408c1e6f10f" (UID: "19e2aba0-8b73-4995-950e-6408c1e6f10f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:36.301754 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.301735 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e2aba0-8b73-4995-950e-6408c1e6f10f-kube-api-access-qkrsz" pod "19e2aba0-8b73-4995-950e-6408c1e6f10f" (UID: "19e2aba0-8b73-4995-950e-6408c1e6f10f"). InnerVolumeSpecName "kube-api-access-qkrsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:36.387232 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.387196 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:36:36.387623 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.387256 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" podUID="d212d906-4b33-44b1-9079-1b5ae0000f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:36:36.401378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.401339 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e2aba0-8b73-4995-950e-6408c1e6f10f-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:36.401378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:36.401377 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkrsz\" (UniqueName: \"kubernetes.io/projected/19e2aba0-8b73-4995-950e-6408c1e6f10f-kube-api-access-qkrsz\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:37.056507 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:37.056469 5838 generic.go:356] "Generic (PLEG): container finished" podID="995c7339-905b-459a-88bf-9971b42ca95b" containerID="4c6cafa548e2a41bddffa3284b7c1704dcc1d4a5220e7d9736f2962137a10f7a" exitCode=0 Oct 30 16:36:37.056697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:37.056557 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" event={"ID":"995c7339-905b-459a-88bf-9971b42ca95b","Type":"ContainerDied","Data":"4c6cafa548e2a41bddffa3284b7c1704dcc1d4a5220e7d9736f2962137a10f7a"} Oct 30 16:36:37.058322 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:37.058294 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" event={"ID":"19e2aba0-8b73-4995-950e-6408c1e6f10f","Type":"ContainerDied","Data":"dcd93df9367d5489ad8b57a6ba0285d6056960c9a79bb5b12705ffd2cdb83de4"} Oct 30 16:36:37.058450 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:37.058344 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcd93df9367d5489ad8b57a6ba0285d6056960c9a79bb5b12705ffd2cdb83de4" Oct 30 16:36:37.058450 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:37.058301 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9" Oct 30 16:36:37.386617 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:37.386532 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:36:37.386617 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:37.386599 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" podUID="d212d906-4b33-44b1-9079-1b5ae0000f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:36:38.371665 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:38.371584 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:38.386850 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:38.386814 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:36:38.387049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:38.386864 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" podUID="d212d906-4b33-44b1-9079-1b5ae0000f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:36:38.485468 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:38.485201 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-xks-ca\") pod \"995c7339-905b-459a-88bf-9971b42ca95b\" (UID: \"995c7339-905b-459a-88bf-9971b42ca95b\") " Oct 30 16:36:38.485468 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:38.485304 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-openshift-ca\") pod \"995c7339-905b-459a-88bf-9971b42ca95b\" (UID: \"995c7339-905b-459a-88bf-9971b42ca95b\") " Oct 30 16:36:38.485468 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:38.485336 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995c7339-905b-459a-88bf-9971b42ca95b-kube-api-access-4m8p6\" (UniqueName: \"kubernetes.io/projected/995c7339-905b-459a-88bf-9971b42ca95b-kube-api-access-4m8p6\") pod \"995c7339-905b-459a-88bf-9971b42ca95b\" (UID: \"995c7339-905b-459a-88bf-9971b42ca95b\") " Oct 30 16:36:38.485771 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:38.485712 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-openshift-ca" pod "995c7339-905b-459a-88bf-9971b42ca95b" (UID: "995c7339-905b-459a-88bf-9971b42ca95b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:38.486372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:38.486106 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:38.498475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:38.497394 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995c7339-905b-459a-88bf-9971b42ca95b-kube-api-access-4m8p6" pod "995c7339-905b-459a-88bf-9971b42ca95b" (UID: "995c7339-905b-459a-88bf-9971b42ca95b"). InnerVolumeSpecName "kube-api-access-4m8p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:38.498889 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:38.498858 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-xks-ca" pod "995c7339-905b-459a-88bf-9971b42ca95b" (UID: "995c7339-905b-459a-88bf-9971b42ca95b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:38.588088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:38.588056 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/995c7339-905b-459a-88bf-9971b42ca95b-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:38.588088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:38.588085 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m8p6\" (UniqueName: \"kubernetes.io/projected/995c7339-905b-459a-88bf-9971b42ca95b-kube-api-access-4m8p6\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:39.098997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:39.098140 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" event={"ID":"995c7339-905b-459a-88bf-9971b42ca95b","Type":"ContainerDied","Data":"6b8c461ea1d8503179af9cba2a7fc7baaba8f034a53bc7edfba32aca22f735c6"} Oct 30 16:36:39.098997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:39.098209 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b8c461ea1d8503179af9cba2a7fc7baaba8f034a53bc7edfba32aca22f735c6" Oct 30 16:36:39.098997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:39.098282 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f" Oct 30 16:36:39.392185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:39.389651 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:36:39.392185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:39.389728 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" podUID="d212d906-4b33-44b1-9079-1b5ae0000f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:36:40.389514 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:40.389470 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:36:40.389726 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:40.389536 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" podUID="d212d906-4b33-44b1-9079-1b5ae0000f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:36:41.387563 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.387529 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:36:41.388218 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.388178 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" podUID="d212d906-4b33-44b1-9079-1b5ae0000f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:36:41.540580 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.540541 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf"] Oct 30 16:36:41.541219 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.541199 5838 cpu_manager.go:414] "containerMap: removing container" podUID="19e2aba0-8b73-4995-950e-6408c1e6f10f" containerName="revoke-keys" Oct 30 16:36:41.541219 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.541218 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e2aba0-8b73-4995-950e-6408c1e6f10f" containerName="revoke-keys" Oct 30 16:36:41.541329 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.541260 5838 cpu_manager.go:414] "containerMap: removing container" podUID="995c7339-905b-459a-88bf-9971b42ca95b" containerName="revoke-keys" Oct 30 16:36:41.541329 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.541266 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995c7339-905b-459a-88bf-9971b42ca95b" containerName="revoke-keys" Oct 30 16:36:41.541329 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.541276 5838 cpu_manager.go:414] "containerMap: removing container" podUID="48a8b00a-4d8e-4428-80eb-b651e333a38f" containerName="revoke-keys" Oct 30 16:36:41.541329 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.541281 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a8b00a-4d8e-4428-80eb-b651e333a38f" containerName="revoke-keys" Oct 30 16:36:41.541440 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.541362 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e2aba0-8b73-4995-950e-6408c1e6f10f" containerName="revoke-keys" Oct 30 16:36:41.541440 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.541378 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="995c7339-905b-459a-88bf-9971b42ca95b" containerName="revoke-keys" Oct 30 16:36:41.541440 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.541384 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a8b00a-4d8e-4428-80eb-b651e333a38f" containerName="revoke-keys" Oct 30 16:36:41.546355 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.546336 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:41.570781 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.570705 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf"] Oct 30 16:36:41.574814 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.574776 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:41.575019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.574818 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:36:41.718108 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.717937 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86b26dcf-gqzkf\" (UID: \"856dbe15-b87c-4c29-9b17-33107352e240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:41.718508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.718477 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86b26dcf-gqzkf\" (UID: \"856dbe15-b87c-4c29-9b17-33107352e240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:41.718694 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.718667 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndcmb\" (UniqueName: \"kubernetes.io/projected/856dbe15-b87c-4c29-9b17-33107352e240-kube-api-access-ndcmb\") pod \"maas-api-revoke-sub-86b26dcf-gqzkf\" (UID: \"856dbe15-b87c-4c29-9b17-33107352e240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:41.819514 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.819482 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86b26dcf-gqzkf\" (UID: \"856dbe15-b87c-4c29-9b17-33107352e240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:41.819684 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.819546 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86b26dcf-gqzkf\" (UID: \"856dbe15-b87c-4c29-9b17-33107352e240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:41.819684 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.819573 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndcmb\" (UniqueName: \"kubernetes.io/projected/856dbe15-b87c-4c29-9b17-33107352e240-kube-api-access-ndcmb\") pod \"maas-api-revoke-sub-86b26dcf-gqzkf\" (UID: \"856dbe15-b87c-4c29-9b17-33107352e240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:41.820459 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.820431 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86b26dcf-gqzkf\" (UID: \"856dbe15-b87c-4c29-9b17-33107352e240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:41.821904 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.821881 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86b26dcf-gqzkf\" (UID: \"856dbe15-b87c-4c29-9b17-33107352e240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:41.843625 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.843590 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndcmb\" (UniqueName: \"kubernetes.io/projected/856dbe15-b87c-4c29-9b17-33107352e240-kube-api-access-ndcmb\") pod \"maas-api-revoke-sub-86b26dcf-gqzkf\" (UID: \"856dbe15-b87c-4c29-9b17-33107352e240\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:41.862706 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:41.862669 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:42.233475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:42.233437 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf"] Oct 30 16:36:42.235732 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:42.235693 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856dbe15_b87c_4c29_9b17_33107352e240.slice/crio-697b5d7f6613f4c542525da4b53685420f08b4c5699ebdf1d01db3b6249add42 WatchSource:0}: Error finding container 697b5d7f6613f4c542525da4b53685420f08b4c5699ebdf1d01db3b6249add42: Status 404 returned error can't find the container with id 697b5d7f6613f4c542525da4b53685420f08b4c5699ebdf1d01db3b6249add42 Oct 30 16:36:42.716869 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:42.716831 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:42.717956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:42.717877 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:42.719435 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:42.719413 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:36:43.138453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:43.138411 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" event={"ID":"856dbe15-b87c-4c29-9b17-33107352e240","Type":"ContainerStarted","Data":"0d6ba2ce285c3177590370ab5c21a447dd8f80806a80431ca2cb2f6d3bd72c93"} Oct 30 16:36:43.138650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:43.138473 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" event={"ID":"856dbe15-b87c-4c29-9b17-33107352e240","Type":"ContainerStarted","Data":"697b5d7f6613f4c542525da4b53685420f08b4c5699ebdf1d01db3b6249add42"} Oct 30 16:36:43.179188 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:43.179132 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" podStartSLOduration=2.179118882 podStartE2EDuration="2.179118882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:43.178268222 +0000 UTC m=+964.926857630" watchObservedRunningTime="2026-10-30 16:36:43.179118882 +0000 UTC m=+964.927708269" Oct 30 16:36:44.187752 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:44.187712 5838 generic.go:356] "Generic (PLEG): container finished" podID="856dbe15-b87c-4c29-9b17-33107352e240" containerID="0d6ba2ce285c3177590370ab5c21a447dd8f80806a80431ca2cb2f6d3bd72c93" exitCode=0 Oct 30 16:36:44.189808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:44.189765 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" event={"ID":"856dbe15-b87c-4c29-9b17-33107352e240","Type":"ContainerDied","Data":"0d6ba2ce285c3177590370ab5c21a447dd8f80806a80431ca2cb2f6d3bd72c93"} Oct 30 16:36:45.788994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:45.761875 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq"] Oct 30 16:36:46.100997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.066553 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq"] Oct 30 16:36:46.143067 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.088230 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:46.162020 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.088285 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 16:36:46.175411 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.161785 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrl2\" (UniqueName: \"kubernetes.io/projected/b01b5da0-655c-4866-b665-5acac0fa0fee-kube-api-access-hsrl2\") pod \"maas-api-revoke-sub-17f819f0-zktnq\" (UID: \"b01b5da0-655c-4866-b665-5acac0fa0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:46.206904 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.179102 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17f819f0-zktnq\" (UID: \"b01b5da0-655c-4866-b665-5acac0fa0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:46.285286 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.285239 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17f819f0-zktnq\" (UID: \"b01b5da0-655c-4866-b665-5acac0fa0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:46.361174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.358074 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:46.445036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.409958 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 16:36:46.445036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.418508 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:46.445036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.422925 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrl2\" (UniqueName: \"kubernetes.io/projected/b01b5da0-655c-4866-b665-5acac0fa0fee-kube-api-access-hsrl2\") pod \"maas-api-revoke-sub-17f819f0-zktnq\" (UID: \"b01b5da0-655c-4866-b665-5acac0fa0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:46.445036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.423015 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17f819f0-zktnq\" (UID: \"b01b5da0-655c-4866-b665-5acac0fa0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:46.445036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.423168 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17f819f0-zktnq\" (UID: \"b01b5da0-655c-4866-b665-5acac0fa0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:46.445036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.432587 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17f819f0-zktnq\" (UID: \"b01b5da0-655c-4866-b665-5acac0fa0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:46.464273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.464231 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:46.469250 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.469219 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:46.510003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.502693 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrl2\" (UniqueName: \"kubernetes.io/projected/b01b5da0-655c-4866-b665-5acac0fa0fee-kube-api-access-hsrl2\") pod \"maas-api-revoke-sub-17f819f0-zktnq\" (UID: \"b01b5da0-655c-4866-b665-5acac0fa0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:46.514883 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.514135 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17f819f0-zktnq\" (UID: \"b01b5da0-655c-4866-b665-5acac0fa0fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:46.550888 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.540066 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856dbe15-b87c-4c29-9b17-33107352e240-kube-api-access-ndcmb\" (UniqueName: \"kubernetes.io/projected/856dbe15-b87c-4c29-9b17-33107352e240-kube-api-access-ndcmb\") pod \"856dbe15-b87c-4c29-9b17-33107352e240\" (UID: \"856dbe15-b87c-4c29-9b17-33107352e240\") " Oct 30 16:36:46.550888 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.540210 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-openshift-ca\") pod \"856dbe15-b87c-4c29-9b17-33107352e240\" (UID: \"856dbe15-b87c-4c29-9b17-33107352e240\") " Oct 30 16:36:46.550888 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.540256 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-xks-ca\") pod \"856dbe15-b87c-4c29-9b17-33107352e240\" (UID: \"856dbe15-b87c-4c29-9b17-33107352e240\") " Oct 30 16:36:46.555471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.555389 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-openshift-ca" pod "856dbe15-b87c-4c29-9b17-33107352e240" (UID: "856dbe15-b87c-4c29-9b17-33107352e240"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:46.604010 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.591066 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856dbe15-b87c-4c29-9b17-33107352e240-kube-api-access-ndcmb" pod "856dbe15-b87c-4c29-9b17-33107352e240" (UID: "856dbe15-b87c-4c29-9b17-33107352e240"). InnerVolumeSpecName "kube-api-access-ndcmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:46.699052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.652851 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:46.762369 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.762335 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndcmb\" (UniqueName: \"kubernetes.io/projected/856dbe15-b87c-4c29-9b17-33107352e240-kube-api-access-ndcmb\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:46.773992 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.687334 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:46.797020 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.720451 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:46.797020 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.687058 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Oct 30 16:36:46.958047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.940083 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 16:36:46.958047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.955696 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:46.977292 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:46.977254 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-xks-ca" pod "856dbe15-b87c-4c29-9b17-33107352e240" (UID: "856dbe15-b87c-4c29-9b17-33107352e240"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:47.007078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.007036 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2"] Oct 30 16:36:47.020070 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.020000 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856dbe15-b87c-4c29-9b17-33107352e240-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:47.020658 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.020628 5838 cpu_manager.go:414] "containerMap: removing container" podUID="856dbe15-b87c-4c29-9b17-33107352e240" containerName="revoke-keys" Oct 30 16:36:47.020863 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.020845 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856dbe15-b87c-4c29-9b17-33107352e240" containerName="revoke-keys" Oct 30 16:36:47.023440 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.023418 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="856dbe15-b87c-4c29-9b17-33107352e240" containerName="revoke-keys" Oct 30 16:36:47.061592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.061559 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:47.068867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.068830 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2"] Oct 30 16:36:47.122831 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.122783 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a67b072-5vcw2\" (UID: \"2019e57f-4469-4d96-882a-88f69dfbb5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:47.123217 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.123191 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br6kx\" (UniqueName: \"kubernetes.io/projected/2019e57f-4469-4d96-882a-88f69dfbb5b3-kube-api-access-br6kx\") pod \"maas-api-revoke-sub-2a67b072-5vcw2\" (UID: \"2019e57f-4469-4d96-882a-88f69dfbb5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:47.123457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.123437 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a67b072-5vcw2\" (UID: \"2019e57f-4469-4d96-882a-88f69dfbb5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:47.269758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.228825 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a67b072-5vcw2\" (UID: \"2019e57f-4469-4d96-882a-88f69dfbb5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:47.269758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.228908 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6kx\" (UniqueName: \"kubernetes.io/projected/2019e57f-4469-4d96-882a-88f69dfbb5b3-kube-api-access-br6kx\") pod \"maas-api-revoke-sub-2a67b072-5vcw2\" (UID: \"2019e57f-4469-4d96-882a-88f69dfbb5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:47.269758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.229035 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a67b072-5vcw2\" (UID: \"2019e57f-4469-4d96-882a-88f69dfbb5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:47.269758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.239372 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a67b072-5vcw2\" (UID: \"2019e57f-4469-4d96-882a-88f69dfbb5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:47.269758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.240074 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a67b072-5vcw2\" (UID: \"2019e57f-4469-4d96-882a-88f69dfbb5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:47.362994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.362001 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br6kx\" (UniqueName: \"kubernetes.io/projected/2019e57f-4469-4d96-882a-88f69dfbb5b3-kube-api-access-br6kx\") pod \"maas-api-revoke-sub-2a67b072-5vcw2\" (UID: \"2019e57f-4469-4d96-882a-88f69dfbb5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:47.510578 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.510544 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:47.559061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.558074 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" event={"ID":"856dbe15-b87c-4c29-9b17-33107352e240","Type":"ContainerDied","Data":"697b5d7f6613f4c542525da4b53685420f08b4c5699ebdf1d01db3b6249add42"} Oct 30 16:36:47.559061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.558140 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697b5d7f6613f4c542525da4b53685420f08b4c5699ebdf1d01db3b6249add42" Oct 30 16:36:47.559061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.558217 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf" Oct 30 16:36:47.684855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:47.684818 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq"] Oct 30 16:36:48.025298 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:48.025269 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2"] Oct 30 16:36:48.026730 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:48.026705 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2019e57f_4469_4d96_882a_88f69dfbb5b3.slice/crio-a9b47a6756bfc2dc74046456b8b6ca2a3df17b5146ebfd60fd313b71bff8e3b5 WatchSource:0}: Error finding container a9b47a6756bfc2dc74046456b8b6ca2a3df17b5146ebfd60fd313b71bff8e3b5: Status 404 returned error can't find the container with id a9b47a6756bfc2dc74046456b8b6ca2a3df17b5146ebfd60fd313b71bff8e3b5 Oct 30 16:36:48.566563 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:48.566476 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" event={"ID":"2019e57f-4469-4d96-882a-88f69dfbb5b3","Type":"ContainerStarted","Data":"c60942bab1ffb4e5ae96ae6db4c87bf41182eb944e4a14436b2dea0f9bbac377"} Oct 30 16:36:48.566563 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:48.566533 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" event={"ID":"2019e57f-4469-4d96-882a-88f69dfbb5b3","Type":"ContainerStarted","Data":"a9b47a6756bfc2dc74046456b8b6ca2a3df17b5146ebfd60fd313b71bff8e3b5"} Oct 30 16:36:48.568163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:48.568138 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" event={"ID":"b01b5da0-655c-4866-b665-5acac0fa0fee","Type":"ContainerStarted","Data":"1a23f0c2a06c8900664355c92733cbe5e9d14ff492d1d63af06bdcff1f9e1bd5"} Oct 30 16:36:48.568293 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:48.568182 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" event={"ID":"b01b5da0-655c-4866-b665-5acac0fa0fee","Type":"ContainerStarted","Data":"7858f27ed6624c5134593fa5c25cd500c7adbbe181445f7324c8f4633b615265"} Oct 30 16:36:48.606959 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:48.606895 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" podStartSLOduration=2.6068773370000002 podStartE2EDuration="2.606877337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:48.604629588 +0000 UTC m=+970.353218986" watchObservedRunningTime="2026-10-30 16:36:48.606877337 +0000 UTC m=+970.355466726" Oct 30 16:36:48.694089 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:48.694026 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" podStartSLOduration=3.694001842 podStartE2EDuration="3.694001842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:48.653263545 +0000 UTC m=+970.401852954" watchObservedRunningTime="2026-10-30 16:36:48.694001842 +0000 UTC m=+970.442591229" Oct 30 16:36:49.577509 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:49.577471 5838 generic.go:356] "Generic (PLEG): container finished" podID="2019e57f-4469-4d96-882a-88f69dfbb5b3" containerID="c60942bab1ffb4e5ae96ae6db4c87bf41182eb944e4a14436b2dea0f9bbac377" exitCode=0 Oct 30 16:36:49.578003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:49.577566 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" event={"ID":"2019e57f-4469-4d96-882a-88f69dfbb5b3","Type":"ContainerDied","Data":"c60942bab1ffb4e5ae96ae6db4c87bf41182eb944e4a14436b2dea0f9bbac377"} Oct 30 16:36:49.579432 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:49.579407 5838 generic.go:356] "Generic (PLEG): container finished" podID="b01b5da0-655c-4866-b665-5acac0fa0fee" containerID="1a23f0c2a06c8900664355c92733cbe5e9d14ff492d1d63af06bdcff1f9e1bd5" exitCode=0 Oct 30 16:36:49.579541 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:49.579485 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" event={"ID":"b01b5da0-655c-4866-b665-5acac0fa0fee","Type":"ContainerDied","Data":"1a23f0c2a06c8900664355c92733cbe5e9d14ff492d1d63af06bdcff1f9e1bd5"} Oct 30 16:36:50.745398 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.745366 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:50.774399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.774374 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:50.794287 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.794258 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-openshift-ca\") pod \"2019e57f-4469-4d96-882a-88f69dfbb5b3\" (UID: \"2019e57f-4469-4d96-882a-88f69dfbb5b3\") " Oct 30 16:36:50.794443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.794347 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01b5da0-655c-4866-b665-5acac0fa0fee-kube-api-access-hsrl2\" (UniqueName: \"kubernetes.io/projected/b01b5da0-655c-4866-b665-5acac0fa0fee-kube-api-access-hsrl2\") pod \"b01b5da0-655c-4866-b665-5acac0fa0fee\" (UID: \"b01b5da0-655c-4866-b665-5acac0fa0fee\") " Oct 30 16:36:50.794443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.794368 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-xks-ca\") pod \"2019e57f-4469-4d96-882a-88f69dfbb5b3\" (UID: \"2019e57f-4469-4d96-882a-88f69dfbb5b3\") " Oct 30 16:36:50.794587 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.794516 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-openshift-ca\") pod \"b01b5da0-655c-4866-b665-5acac0fa0fee\" (UID: \"b01b5da0-655c-4866-b665-5acac0fa0fee\") " Oct 30 16:36:50.794587 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.794547 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2019e57f-4469-4d96-882a-88f69dfbb5b3-kube-api-access-br6kx\" (UniqueName: \"kubernetes.io/projected/2019e57f-4469-4d96-882a-88f69dfbb5b3-kube-api-access-br6kx\") pod \"2019e57f-4469-4d96-882a-88f69dfbb5b3\" (UID: \"2019e57f-4469-4d96-882a-88f69dfbb5b3\") " Oct 30 16:36:50.794700 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.794612 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-xks-ca\") pod \"b01b5da0-655c-4866-b665-5acac0fa0fee\" (UID: \"b01b5da0-655c-4866-b665-5acac0fa0fee\") " Oct 30 16:36:50.794700 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.794662 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-openshift-ca" pod "2019e57f-4469-4d96-882a-88f69dfbb5b3" (UID: "2019e57f-4469-4d96-882a-88f69dfbb5b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:50.795194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.795058 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-openshift-ca" pod "b01b5da0-655c-4866-b665-5acac0fa0fee" (UID: "b01b5da0-655c-4866-b665-5acac0fa0fee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:50.795318 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.795192 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:50.795318 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.795223 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:50.797258 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.797230 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-xks-ca" pod "2019e57f-4469-4d96-882a-88f69dfbb5b3" (UID: "2019e57f-4469-4d96-882a-88f69dfbb5b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:50.797474 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.797447 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01b5da0-655c-4866-b665-5acac0fa0fee-kube-api-access-hsrl2" pod "b01b5da0-655c-4866-b665-5acac0fa0fee" (UID: "b01b5da0-655c-4866-b665-5acac0fa0fee"). InnerVolumeSpecName "kube-api-access-hsrl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:50.797585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.797487 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-xks-ca" pod "b01b5da0-655c-4866-b665-5acac0fa0fee" (UID: "b01b5da0-655c-4866-b665-5acac0fa0fee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:50.798305 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.798278 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2019e57f-4469-4d96-882a-88f69dfbb5b3-kube-api-access-br6kx" pod "2019e57f-4469-4d96-882a-88f69dfbb5b3" (UID: "2019e57f-4469-4d96-882a-88f69dfbb5b3"). InnerVolumeSpecName "kube-api-access-br6kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:50.896025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.895937 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsrl2\" (UniqueName: \"kubernetes.io/projected/b01b5da0-655c-4866-b665-5acac0fa0fee-kube-api-access-hsrl2\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:50.896025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.895965 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2019e57f-4469-4d96-882a-88f69dfbb5b3-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:50.896025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.895997 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br6kx\" (UniqueName: \"kubernetes.io/projected/2019e57f-4469-4d96-882a-88f69dfbb5b3-kube-api-access-br6kx\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:50.896025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:50.896006 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b01b5da0-655c-4866-b665-5acac0fa0fee-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:51.607331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.607299 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" Oct 30 16:36:51.607591 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.607306 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2" event={"ID":"2019e57f-4469-4d96-882a-88f69dfbb5b3","Type":"ContainerDied","Data":"a9b47a6756bfc2dc74046456b8b6ca2a3df17b5146ebfd60fd313b71bff8e3b5"} Oct 30 16:36:51.607678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.607668 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b47a6756bfc2dc74046456b8b6ca2a3df17b5146ebfd60fd313b71bff8e3b5" Oct 30 16:36:51.609008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.608963 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" event={"ID":"b01b5da0-655c-4866-b665-5acac0fa0fee","Type":"ContainerDied","Data":"7858f27ed6624c5134593fa5c25cd500c7adbbe181445f7324c8f4633b615265"} Oct 30 16:36:51.609008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.608998 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq" Oct 30 16:36:51.609151 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.609028 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7858f27ed6624c5134593fa5c25cd500c7adbbe181445f7324c8f4633b615265" Oct 30 16:36:51.862198 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.862120 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf"] Oct 30 16:36:51.862729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.862715 5838 cpu_manager.go:414] "containerMap: removing container" podUID="b01b5da0-655c-4866-b665-5acac0fa0fee" containerName="revoke-keys" Oct 30 16:36:51.862770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.862730 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01b5da0-655c-4866-b665-5acac0fa0fee" containerName="revoke-keys" Oct 30 16:36:51.862770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.862749 5838 cpu_manager.go:414] "containerMap: removing container" podUID="2019e57f-4469-4d96-882a-88f69dfbb5b3" containerName="revoke-keys" Oct 30 16:36:51.862770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.862753 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2019e57f-4469-4d96-882a-88f69dfbb5b3" containerName="revoke-keys" Oct 30 16:36:51.862861 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.862841 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01b5da0-655c-4866-b665-5acac0fa0fee" containerName="revoke-keys" Oct 30 16:36:51.862861 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.862856 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="2019e57f-4469-4d96-882a-88f69dfbb5b3" containerName="revoke-keys" Oct 30 16:36:51.866777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.866755 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:51.881241 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.881215 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:51.881435 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.881421 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:36:51.886189 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.886128 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf"] Oct 30 16:36:51.908749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.908710 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afe785e9-95dxf\" (UID: \"d2807e2f-a164-4293-bf8b-4461980d31ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:51.908917 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.908777 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84sdl\" (UniqueName: \"kubernetes.io/projected/d2807e2f-a164-4293-bf8b-4461980d31ec-kube-api-access-84sdl\") pod \"maas-api-revoke-sub-afe785e9-95dxf\" (UID: \"d2807e2f-a164-4293-bf8b-4461980d31ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:51.908917 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:51.908853 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afe785e9-95dxf\" (UID: \"d2807e2f-a164-4293-bf8b-4461980d31ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:52.009829 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:52.009794 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afe785e9-95dxf\" (UID: \"d2807e2f-a164-4293-bf8b-4461980d31ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:52.010054 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:52.009851 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84sdl\" (UniqueName: \"kubernetes.io/projected/d2807e2f-a164-4293-bf8b-4461980d31ec-kube-api-access-84sdl\") pod \"maas-api-revoke-sub-afe785e9-95dxf\" (UID: \"d2807e2f-a164-4293-bf8b-4461980d31ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:52.010054 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:52.009879 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afe785e9-95dxf\" (UID: \"d2807e2f-a164-4293-bf8b-4461980d31ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:52.010668 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:52.010638 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afe785e9-95dxf\" (UID: \"d2807e2f-a164-4293-bf8b-4461980d31ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:52.012264 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:52.012244 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afe785e9-95dxf\" (UID: \"d2807e2f-a164-4293-bf8b-4461980d31ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:52.033155 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:52.033115 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84sdl\" (UniqueName: \"kubernetes.io/projected/d2807e2f-a164-4293-bf8b-4461980d31ec-kube-api-access-84sdl\") pod \"maas-api-revoke-sub-afe785e9-95dxf\" (UID: \"d2807e2f-a164-4293-bf8b-4461980d31ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:52.181710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:52.181621 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:52.551387 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:52.551363 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf"] Oct 30 16:36:52.552796 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:36:52.552768 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2807e2f_a164_4293_bf8b_4461980d31ec.slice/crio-93504e51f5dbc3fcad11544372787965cb6ab14fa4e981e535ebf12528b3cd0d WatchSource:0}: Error finding container 93504e51f5dbc3fcad11544372787965cb6ab14fa4e981e535ebf12528b3cd0d: Status 404 returned error can't find the container with id 93504e51f5dbc3fcad11544372787965cb6ab14fa4e981e535ebf12528b3cd0d Oct 30 16:36:52.619040 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:52.619000 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" event={"ID":"d2807e2f-a164-4293-bf8b-4461980d31ec","Type":"ContainerStarted","Data":"93504e51f5dbc3fcad11544372787965cb6ab14fa4e981e535ebf12528b3cd0d"} Oct 30 16:36:53.629282 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:53.629192 5838 generic.go:356] "Generic (PLEG): container finished" podID="d2807e2f-a164-4293-bf8b-4461980d31ec" containerID="f4130a9bf6a46d00fffe0c7636be1ec0bf5443b16a28bcafec92801ef34839d7" exitCode=0 Oct 30 16:36:53.629282 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:53.629231 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" event={"ID":"d2807e2f-a164-4293-bf8b-4461980d31ec","Type":"ContainerDied","Data":"f4130a9bf6a46d00fffe0c7636be1ec0bf5443b16a28bcafec92801ef34839d7"} Oct 30 16:36:54.790502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:54.790473 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:54.860748 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:54.860710 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-xks-ca\") pod \"d2807e2f-a164-4293-bf8b-4461980d31ec\" (UID: \"d2807e2f-a164-4293-bf8b-4461980d31ec\") " Oct 30 16:36:54.860943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:54.860771 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2807e2f-a164-4293-bf8b-4461980d31ec-kube-api-access-84sdl\" (UniqueName: \"kubernetes.io/projected/d2807e2f-a164-4293-bf8b-4461980d31ec-kube-api-access-84sdl\") pod \"d2807e2f-a164-4293-bf8b-4461980d31ec\" (UID: \"d2807e2f-a164-4293-bf8b-4461980d31ec\") " Oct 30 16:36:54.860943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:54.860842 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-openshift-ca\") pod \"d2807e2f-a164-4293-bf8b-4461980d31ec\" (UID: \"d2807e2f-a164-4293-bf8b-4461980d31ec\") " Oct 30 16:36:54.861355 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:54.861323 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-openshift-ca" pod "d2807e2f-a164-4293-bf8b-4461980d31ec" (UID: "d2807e2f-a164-4293-bf8b-4461980d31ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:54.861482 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:54.861444 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:54.863502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:54.863471 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-xks-ca" pod "d2807e2f-a164-4293-bf8b-4461980d31ec" (UID: "d2807e2f-a164-4293-bf8b-4461980d31ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:54.863597 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:54.863566 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2807e2f-a164-4293-bf8b-4461980d31ec-kube-api-access-84sdl" pod "d2807e2f-a164-4293-bf8b-4461980d31ec" (UID: "d2807e2f-a164-4293-bf8b-4461980d31ec"). InnerVolumeSpecName "kube-api-access-84sdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:54.962783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:54.962693 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2807e2f-a164-4293-bf8b-4461980d31ec-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:54.962783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:54.962731 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84sdl\" (UniqueName: \"kubernetes.io/projected/d2807e2f-a164-4293-bf8b-4461980d31ec-kube-api-access-84sdl\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:36:55.646271 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:55.646241 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" Oct 30 16:36:55.646454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:55.646267 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf" event={"ID":"d2807e2f-a164-4293-bf8b-4461980d31ec","Type":"ContainerDied","Data":"93504e51f5dbc3fcad11544372787965cb6ab14fa4e981e535ebf12528b3cd0d"} Oct 30 16:36:55.646454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:55.646322 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93504e51f5dbc3fcad11544372787965cb6ab14fa4e981e535ebf12528b3cd0d" Oct 30 16:36:56.200142 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:56.200105 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:36:59.339993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:59.339100 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:36:59.408003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:59.407054 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:59.458157 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:59.458105 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:59.478307 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:59.478259 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:59.707055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:59.706117 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:59.896244 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:36:59.895098 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:01.801327 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:37:01.801283 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:37:02.164650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.164572 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr"] Oct 30 16:37:02.165560 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.165524 5838 cpu_manager.go:414] "containerMap: removing container" podUID="d2807e2f-a164-4293-bf8b-4461980d31ec" containerName="revoke-keys" Oct 30 16:37:02.165560 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.165551 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2807e2f-a164-4293-bf8b-4461980d31ec" containerName="revoke-keys" Oct 30 16:37:02.165773 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.165713 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2807e2f-a164-4293-bf8b-4461980d31ec" containerName="revoke-keys" Oct 30 16:37:02.170165 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.170140 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.187121 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.187094 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr"] Oct 30 16:37:02.193684 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.193658 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ccf5c9\"/\"distinct-w3-ccf5c9-kserve-self-signed-certs\"" Oct 30 16:37:02.248558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.248520 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1aa2186-e285-4d17-b26a-e17f9f321266-tls-certs\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.248755 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.248598 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-dshm\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.248755 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.248727 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glp6h\" (UniqueName: \"kubernetes.io/projected/d1aa2186-e285-4d17-b26a-e17f9f321266-kube-api-access-glp6h\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.248894 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.248878 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-tmp-dir\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.249014 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.248997 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-model-cache\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.249056 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.249046 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-home\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.366641 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.366601 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-home\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.367395 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.367371 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1aa2186-e285-4d17-b26a-e17f9f321266-tls-certs\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.381999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.368056 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-dshm\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.384848 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.384818 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glp6h\" (UniqueName: \"kubernetes.io/projected/d1aa2186-e285-4d17-b26a-e17f9f321266-kube-api-access-glp6h\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.385503 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.385477 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-tmp-dir\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.386103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.386083 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-model-cache\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.414370 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.367263 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-home\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.414370 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.385919 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-tmp-dir\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.414370 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.383134 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1aa2186-e285-4d17-b26a-e17f9f321266-tls-certs\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.414370 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.391213 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-dshm\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.414370 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.391453 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-model-cache\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.450052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.432246 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glp6h\" (UniqueName: \"kubernetes.io/projected/d1aa2186-e285-4d17-b26a-e17f9f321266-kube-api-access-glp6h\") pod \"distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.496334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.496295 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:02.810503 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.810471 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr"] Oct 30 16:37:02.811757 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:02.811731 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1aa2186_e285_4d17_b26a_e17f9f321266.slice/crio-9623bd85b53d0dc44c84ba2bf6d4b6eae3e93964c08ec2eb284053a278be9808 WatchSource:0}: Error finding container 9623bd85b53d0dc44c84ba2bf6d4b6eae3e93964c08ec2eb284053a278be9808: Status 404 returned error can't find the container with id 9623bd85b53d0dc44c84ba2bf6d4b6eae3e93964c08ec2eb284053a278be9808 Oct 30 16:37:02.904581 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.904545 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" event={"ID":"d1aa2186-e285-4d17-b26a-e17f9f321266","Type":"ContainerStarted","Data":"bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49"} Oct 30 16:37:02.904772 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.904595 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" event={"ID":"d1aa2186-e285-4d17-b26a-e17f9f321266","Type":"ContainerStarted","Data":"9623bd85b53d0dc44c84ba2bf6d4b6eae3e93964c08ec2eb284053a278be9808"} Oct 30 16:37:02.948260 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:02.948208 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podStartSLOduration=0.948193788 podStartE2EDuration="948.193788ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:02.946547295 +0000 UTC m=+984.695136702" watchObservedRunningTime="2026-10-30 16:37:02.948193788 +0000 UTC m=+984.696783181" Oct 30 16:37:03.376167 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.376128 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d"] Oct 30 16:37:03.381718 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.381693 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.411025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.405038 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d"] Oct 30 16:37:03.422042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.421815 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-podinfo\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.422042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.421857 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ad2f35-f82b-43d6-b068-163cc3192b26-istiod-ca-cert\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.422042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.421890 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.422042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.421952 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-credential-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.422042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.422031 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwjks\" (UniqueName: \"kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-kube-api-access-mwjks\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.422399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.422087 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-data\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.422399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.422128 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-certs\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.422399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.422197 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-token\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.422399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.422249 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-envoy\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.497177 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.497142 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:03.523984 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.523941 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-envoy\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.524151 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.524130 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-podinfo\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.524194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.524164 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ad2f35-f82b-43d6-b068-163cc3192b26-istiod-ca-cert\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.524258 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.524195 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.524308 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.524266 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-credential-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.524363 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.524313 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwjks\" (UniqueName: \"kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-kube-api-access-mwjks\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.524705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.524679 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.524842 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.524774 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-credential-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.524842 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.524796 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-data\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.525027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.524851 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-certs\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.525027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.524947 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-token\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.525138 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.525025 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ad2f35-f82b-43d6-b068-163cc3192b26-istiod-ca-cert\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.525205 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.525165 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-data\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.525296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.525274 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-certs\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.526381 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.526360 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-envoy\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.526791 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.526773 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-podinfo\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.554803 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.554772 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-token\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.554936 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.554903 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwjks\" (UniqueName: \"kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-kube-api-access-mwjks\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.561935 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.561909 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v"] Oct 30 16:37:03.567436 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.567417 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.591358 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.591325 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v"] Oct 30 16:37:03.626699 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.626526 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a2dbaee-1611-4433-b37c-90487d456baf-istio-podinfo\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.626699 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.626599 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-data\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.626954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.626702 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-certs\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.626954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.626769 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a2dbaee-1611-4433-b37c-90487d456baf-istiod-ca-cert\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.626954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.626801 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-envoy\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.626954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.626855 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz47h\" (UniqueName: \"kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-kube-api-access-sz47h\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.627174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.627018 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-credential-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.627174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.627064 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-istio-token\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.627174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.627085 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.699560 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.699515 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:03.731963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.730534 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a2dbaee-1611-4433-b37c-90487d456baf-istio-podinfo\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.731963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.730612 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-data\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.731963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.730670 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-certs\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.731963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.730733 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a2dbaee-1611-4433-b37c-90487d456baf-istiod-ca-cert\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.731963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.730764 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-envoy\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.731963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.730805 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz47h\" (UniqueName: \"kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-kube-api-access-sz47h\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.731963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.730867 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-credential-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.731963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.730921 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-istio-token\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.731963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.730990 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.738349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.737329 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-certs\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.738349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.737631 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-data\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.738349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.738033 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-credential-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.738349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.738176 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a2dbaee-1611-4433-b37c-90487d456baf-istiod-ca-cert\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.744762 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.741667 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.744762 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.744386 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-envoy\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.746940 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.746576 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a2dbaee-1611-4433-b37c-90487d456baf-istio-podinfo\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.760853 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.760829 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-istio-token\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.764586 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.764561 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz47h\" (UniqueName: \"kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-kube-api-access-sz47h\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:03.893169 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:03.893078 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:04.071860 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.071829 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d"] Oct 30 16:37:04.072124 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:04.072104 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ad2f35_f82b_43d6_b068_163cc3192b26.slice/crio-0671a403bbba6cde40913224e1290d1ebf7af05407ce6b2a6b0b686a92403355 WatchSource:0}: Error finding container 0671a403bbba6cde40913224e1290d1ebf7af05407ce6b2a6b0b686a92403355: Status 404 returned error can't find the container with id 0671a403bbba6cde40913224e1290d1ebf7af05407ce6b2a6b0b686a92403355 Oct 30 16:37:04.074508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.074477 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:04.074608 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.074538 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:04.074608 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.074566 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:04.099372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.099340 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v"] Oct 30 16:37:04.105138 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.105116 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.130740 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.130704 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v"] Oct 30 16:37:04.141898 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.141849 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-podinfo\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.141898 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.141898 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-token\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.142134 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.142020 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istiod-ca-cert\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.142196 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.142149 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-certs\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.142280 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.142265 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-data\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.142349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.142325 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fzpd\" (UniqueName: \"kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-kube-api-access-6fzpd\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.142449 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.142428 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-envoy\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.142679 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.142475 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.142679 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.142526 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-credential-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.220495 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.220462 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v"] Oct 30 16:37:04.220953 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:04.220928 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2dbaee_1611_4433_b37c_90487d456baf.slice/crio-190119d76db8d4a7862bb407562704cede792f02b9b3ad1411d5dbbd6a72e7c9 WatchSource:0}: Error finding container 190119d76db8d4a7862bb407562704cede792f02b9b3ad1411d5dbbd6a72e7c9: Status 404 returned error can't find the container with id 190119d76db8d4a7862bb407562704cede792f02b9b3ad1411d5dbbd6a72e7c9 Oct 30 16:37:04.223629 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.223587 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:04.223724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.223649 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:04.223724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.223680 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:04.244146 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.244112 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-data\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.244269 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.244165 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fzpd\" (UniqueName: \"kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-kube-api-access-6fzpd\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.244269 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.244256 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-envoy\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.244334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.244298 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.244384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.244346 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-credential-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.244490 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.244471 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-podinfo\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.244557 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.244513 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-token\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.244557 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.244537 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istiod-ca-cert\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.244663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.244609 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-certs\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.245081 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.245056 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.245221 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.245142 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-data\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.245672 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.245086 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-credential-socket\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.245928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.245906 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istiod-ca-cert\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.246453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.246431 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-certs\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.247093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.247064 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-envoy\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.248996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.248963 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-podinfo\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.266964 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.266938 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fzpd\" (UniqueName: \"kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-kube-api-access-6fzpd\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.268277 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.268242 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-token\") pod \"e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.446950 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.446856 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:04.791534 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.791500 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v"] Oct 30 16:37:04.792115 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:04.792082 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439bafeb_d93c_46f1_a33f_1a1ffcd1d776.slice/crio-ec00d6d46cd6c70f5a9c69017daf3af6e72869fbf95f7a6f4149360f3a10ef1a WatchSource:0}: Error finding container ec00d6d46cd6c70f5a9c69017daf3af6e72869fbf95f7a6f4149360f3a10ef1a: Status 404 returned error can't find the container with id ec00d6d46cd6c70f5a9c69017daf3af6e72869fbf95f7a6f4149360f3a10ef1a Oct 30 16:37:04.794707 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.794674 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:04.795282 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.795243 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:04.795530 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.795503 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:04.980174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.978467 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" event={"ID":"5a2dbaee-1611-4433-b37c-90487d456baf","Type":"ContainerStarted","Data":"d5bf51754a7e8e256adfbc38a78352ac17e58ec1818db70bc955d12c3cf1c121"} Oct 30 16:37:04.980174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.978562 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" event={"ID":"5a2dbaee-1611-4433-b37c-90487d456baf","Type":"ContainerStarted","Data":"190119d76db8d4a7862bb407562704cede792f02b9b3ad1411d5dbbd6a72e7c9"} Oct 30 16:37:04.980556 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.980266 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" event={"ID":"439bafeb-d93c-46f1-a33f-1a1ffcd1d776","Type":"ContainerStarted","Data":"ec00d6d46cd6c70f5a9c69017daf3af6e72869fbf95f7a6f4149360f3a10ef1a"} Oct 30 16:37:04.982698 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.982613 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" event={"ID":"63ad2f35-f82b-43d6-b068-163cc3192b26","Type":"ContainerStarted","Data":"b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d"} Oct 30 16:37:04.982698 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.982664 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" event={"ID":"63ad2f35-f82b-43d6-b068-163cc3192b26","Type":"ContainerStarted","Data":"0671a403bbba6cde40913224e1290d1ebf7af05407ce6b2a6b0b686a92403355"} Oct 30 16:37:04.986663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.986641 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7"] Oct 30 16:37:04.990985 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:04.990956 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:05.008662 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.008599 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7"] Oct 30 16:37:05.012226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.012199 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:37:05.012378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.012243 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:37:05.179040 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.178323 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" podStartSLOduration=2.178302804 podStartE2EDuration="2.178302804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:05.074853499 +0000 UTC m=+986.823442912" watchObservedRunningTime="2026-10-30 16:37:05.178302804 +0000 UTC m=+986.926892192" Oct 30 16:37:05.183238 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.183189 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" podStartSLOduration=2.18317619 podStartE2EDuration="2.18317619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:05.157427331 +0000 UTC m=+986.906016738" watchObservedRunningTime="2026-10-30 16:37:05.18317619 +0000 UTC m=+986.931765643" Oct 30 16:37:05.188870 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.188841 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmbgn\" (UniqueName: \"kubernetes.io/projected/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-kube-api-access-qmbgn\") pod \"maas-api-revoke-sub-74447167-w5hw7\" (UID: \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:05.189611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.189580 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74447167-w5hw7\" (UID: \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:05.189674 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.189638 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74447167-w5hw7\" (UID: \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:05.292376 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.292325 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74447167-w5hw7\" (UID: \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:05.292550 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.292384 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74447167-w5hw7\" (UID: \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:05.292550 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.292492 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmbgn\" (UniqueName: \"kubernetes.io/projected/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-kube-api-access-qmbgn\") pod \"maas-api-revoke-sub-74447167-w5hw7\" (UID: \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:05.293356 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.293335 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74447167-w5hw7\" (UID: \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:05.295590 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.295568 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74447167-w5hw7\" (UID: \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:05.352223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.352186 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmbgn\" (UniqueName: \"kubernetes.io/projected/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-kube-api-access-qmbgn\") pod \"maas-api-revoke-sub-74447167-w5hw7\" (UID: \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:05.605366 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.605330 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:05.703345 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.702657 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:05.707418 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.707069 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:37:05.707418 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.707133 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" podUID="63ad2f35-f82b-43d6-b068-163cc3192b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:37:05.952145 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.913335 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:05.963046 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.952410 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:37:05.963046 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:05.952464 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" podUID="5a2dbaee-1611-4433-b37c-90487d456baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:37:06.210760 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:06.210681 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" event={"ID":"439bafeb-d93c-46f1-a33f-1a1ffcd1d776","Type":"ContainerStarted","Data":"385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17"} Oct 30 16:37:06.462393 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:06.462224 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:06.466655 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:06.466564 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:37:06.466913 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:06.466885 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 16:37:06.490999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:06.480383 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" podStartSLOduration=2.480359172 podStartE2EDuration="2.480359172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:06.29339528 +0000 UTC m=+988.041984704" watchObservedRunningTime="2026-10-30 16:37:06.480359172 +0000 UTC m=+988.228948560" Oct 30 16:37:06.490999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:06.481158 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7"] Oct 30 16:37:06.499666 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:06.499497 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d56ec6_bce3_46c4_b471_e1c5cfe006e4.slice/crio-ed9bb5bf29e59c7021df541e8bd2af4859d25ea3d26745714741526d84caeabe WatchSource:0}: Error finding container ed9bb5bf29e59c7021df541e8bd2af4859d25ea3d26745714741526d84caeabe: Status 404 returned error can't find the container with id ed9bb5bf29e59c7021df541e8bd2af4859d25ea3d26745714741526d84caeabe Oct 30 16:37:06.701508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:06.701471 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:37:06.701710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:06.701532 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" podUID="63ad2f35-f82b-43d6-b068-163cc3192b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:37:06.894254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:06.893829 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:37:06.894254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:06.893899 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" podUID="5a2dbaee-1611-4433-b37c-90487d456baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:37:07.295293 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:07.295235 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" event={"ID":"84d56ec6-bce3-46c4-b471-e1c5cfe006e4","Type":"ContainerStarted","Data":"737ce2dcfea75c397dd118e2f9cfeafd3b87d03f42dccd45e32f9736ee9706dd"} Oct 30 16:37:07.295940 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:07.295912 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" event={"ID":"84d56ec6-bce3-46c4-b471-e1c5cfe006e4","Type":"ContainerStarted","Data":"ed9bb5bf29e59c7021df541e8bd2af4859d25ea3d26745714741526d84caeabe"} Oct 30 16:37:07.336482 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:07.336408 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" podStartSLOduration=3.336389133 podStartE2EDuration="3.336389133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:07.335406896 +0000 UTC m=+989.083996284" watchObservedRunningTime="2026-10-30 16:37:07.336389133 +0000 UTC m=+989.084978521" Oct 30 16:37:07.468999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:07.466092 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:37:07.468999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:07.466157 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 16:37:07.700467 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:07.700427 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:37:07.700635 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:07.700482 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" podUID="63ad2f35-f82b-43d6-b068-163cc3192b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:37:07.893654 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:07.893571 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:37:07.893654 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:07.893632 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" podUID="5a2dbaee-1611-4433-b37c-90487d456baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:37:08.506010 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:08.505458 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:37:08.506010 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:08.505523 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 16:37:08.523045 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:08.507008 5838 generic.go:356] "Generic (PLEG): container finished" podID="84d56ec6-bce3-46c4-b471-e1c5cfe006e4" containerID="737ce2dcfea75c397dd118e2f9cfeafd3b87d03f42dccd45e32f9736ee9706dd" exitCode=0 Oct 30 16:37:08.523045 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:08.508498 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" event={"ID":"84d56ec6-bce3-46c4-b471-e1c5cfe006e4","Type":"ContainerDied","Data":"737ce2dcfea75c397dd118e2f9cfeafd3b87d03f42dccd45e32f9736ee9706dd"} Oct 30 16:37:08.700738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:08.700702 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:37:08.700995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:08.700957 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" podUID="63ad2f35-f82b-43d6-b068-163cc3192b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:37:08.894081 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:08.893716 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:37:08.894081 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:08.893775 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" podUID="5a2dbaee-1611-4433-b37c-90487d456baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:37:09.449538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.449481 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:37:09.449807 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.449778 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 16:37:09.592998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.590080 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d"] Oct 30 16:37:09.626554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.621281 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d"] Oct 30 16:37:09.626554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.621446 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:09.705009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.704893 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:37:09.707324 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.706070 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" podUID="63ad2f35-f82b-43d6-b068-163cc3192b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:37:09.802831 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.801710 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47170442-ndz4d\" (UID: \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:09.802831 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.801780 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47170442-ndz4d\" (UID: \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:09.803872 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.803841 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwsqn\" (UniqueName: \"kubernetes.io/projected/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-kube-api-access-gwsqn\") pod \"maas-api-revoke-sub-47170442-ndz4d\" (UID: \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:09.897992 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.895354 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:37:09.897992 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.895415 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" podUID="5a2dbaee-1611-4433-b37c-90487d456baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:37:09.905813 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.905774 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47170442-ndz4d\" (UID: \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:09.906023 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.906006 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47170442-ndz4d\" (UID: \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:09.907992 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.907584 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwsqn\" (UniqueName: \"kubernetes.io/projected/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-kube-api-access-gwsqn\") pod \"maas-api-revoke-sub-47170442-ndz4d\" (UID: \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:09.908906 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.908709 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47170442-ndz4d\" (UID: \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:09.913149 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.913123 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47170442-ndz4d\" (UID: \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:09.960073 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:09.956123 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwsqn\" (UniqueName: \"kubernetes.io/projected/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-kube-api-access-gwsqn\") pod \"maas-api-revoke-sub-47170442-ndz4d\" (UID: \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:10.008090 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.007772 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:10.203460 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.203429 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:10.211400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.211329 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-openshift-ca\") pod \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\" (UID: \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\") " Oct 30 16:37:10.211400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.211384 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-kube-api-access-qmbgn\" (UniqueName: \"kubernetes.io/projected/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-kube-api-access-qmbgn\") pod \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\" (UID: \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\") " Oct 30 16:37:10.211552 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.211418 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-xks-ca\") pod \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\" (UID: \"84d56ec6-bce3-46c4-b471-e1c5cfe006e4\") " Oct 30 16:37:10.212503 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.212478 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-openshift-ca" pod "84d56ec6-bce3-46c4-b471-e1c5cfe006e4" (UID: "84d56ec6-bce3-46c4-b471-e1c5cfe006e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:10.213895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.213864 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-xks-ca" pod "84d56ec6-bce3-46c4-b471-e1c5cfe006e4" (UID: "84d56ec6-bce3-46c4-b471-e1c5cfe006e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:10.231996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.220146 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-kube-api-access-qmbgn" pod "84d56ec6-bce3-46c4-b471-e1c5cfe006e4" (UID: "84d56ec6-bce3-46c4-b471-e1c5cfe006e4"). InnerVolumeSpecName "kube-api-access-qmbgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:10.356998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.313743 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:10.356998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.313780 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmbgn\" (UniqueName: \"kubernetes.io/projected/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-kube-api-access-qmbgn\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:10.356998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.313796 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84d56ec6-bce3-46c4-b471-e1c5cfe006e4-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:10.465624 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.465107 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:37:10.465624 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.465173 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 16:37:10.588824 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.588783 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" event={"ID":"84d56ec6-bce3-46c4-b471-e1c5cfe006e4","Type":"ContainerDied","Data":"ed9bb5bf29e59c7021df541e8bd2af4859d25ea3d26745714741526d84caeabe"} Oct 30 16:37:10.589012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.588862 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed9bb5bf29e59c7021df541e8bd2af4859d25ea3d26745714741526d84caeabe" Oct 30 16:37:10.589012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.588941 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7" Oct 30 16:37:10.700896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.700859 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:37:10.701312 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.700927 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" podUID="63ad2f35-f82b-43d6-b068-163cc3192b26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:37:10.857669 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.857627 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d"] Oct 30 16:37:10.897303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.897270 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:37:10.898996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:10.897527 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" podUID="5a2dbaee-1611-4433-b37c-90487d456baf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:37:11.450411 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:11.449934 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:37:11.450411 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:11.450017 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 16:37:11.629689 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:11.629652 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" event={"ID":"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5","Type":"ContainerStarted","Data":"ed82d6e13b018f6741b2eb65ebd719725086018806971fe7bd26ec69b0778c15"} Oct 30 16:37:11.706927 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:11.706887 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:11.707443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:11.707417 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:11.708401 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:11.708374 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:37:12.317505 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:12.317471 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:12.318497 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:12.318465 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:12.320340 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:12.320316 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:37:12.447737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:12.447624 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 16:37:12.447737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:12.447693 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 16:37:12.496906 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:12.496853 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:12.511163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:12.511125 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:12.641262 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:12.641223 5838 generic.go:356] "Generic (PLEG): container finished" podID="21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5" containerID="47031d2482b741a596bb9ed448d7dd7707fe4fd89f1cfd4a01312c1e2dca1493" exitCode=0 Oct 30 16:37:12.641508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:12.641272 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" event={"ID":"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5","Type":"ContainerDied","Data":"47031d2482b741a596bb9ed448d7dd7707fe4fd89f1cfd4a01312c1e2dca1493"} Oct 30 16:37:12.653687 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:12.653655 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:13.451571 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.451541 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:13.451979 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.451939 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:13.453110 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.453091 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:37:13.806587 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.806557 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:13.849991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.848204 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-openshift-ca\") pod \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\" (UID: \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\") " Oct 30 16:37:13.849991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.848258 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-kube-api-access-gwsqn\" (UniqueName: \"kubernetes.io/projected/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-kube-api-access-gwsqn\") pod \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\" (UID: \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\") " Oct 30 16:37:13.849991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.848492 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-xks-ca\") pod \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\" (UID: \"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5\") " Oct 30 16:37:13.857995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.857477 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-openshift-ca" pod "21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5" (UID: "21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:13.890555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.890493 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-kube-api-access-gwsqn" pod "21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5" (UID: "21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5"). InnerVolumeSpecName "kube-api-access-gwsqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:13.890866 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.890824 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-xks-ca" pod "21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5" (UID: "21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:13.961055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.960049 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:13.961055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.960087 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwsqn\" (UniqueName: \"kubernetes.io/projected/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-kube-api-access-gwsqn\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:13.961055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:13.960104 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:14.693333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:14.693286 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" Oct 30 16:37:14.711021 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:14.700349 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d" event={"ID":"21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5","Type":"ContainerDied","Data":"ed82d6e13b018f6741b2eb65ebd719725086018806971fe7bd26ec69b0778c15"} Oct 30 16:37:14.711021 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:14.700434 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed82d6e13b018f6741b2eb65ebd719725086018806971fe7bd26ec69b0778c15" Oct 30 16:37:18.438078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.438041 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg"] Oct 30 16:37:18.440260 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.440229 5838 cpu_manager.go:414] "containerMap: removing container" podUID="84d56ec6-bce3-46c4-b471-e1c5cfe006e4" containerName="revoke-keys" Oct 30 16:37:18.440444 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.440429 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d56ec6-bce3-46c4-b471-e1c5cfe006e4" containerName="revoke-keys" Oct 30 16:37:18.440625 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.440613 5838 cpu_manager.go:414] "containerMap: removing container" podUID="21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5" containerName="revoke-keys" Oct 30 16:37:18.440703 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.440693 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5" containerName="revoke-keys" Oct 30 16:37:18.441118 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.441098 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5" containerName="revoke-keys" Oct 30 16:37:18.441254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.441243 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d56ec6-bce3-46c4-b471-e1c5cfe006e4" containerName="revoke-keys" Oct 30 16:37:18.447237 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.447205 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.466496 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.466454 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg"] Oct 30 16:37:18.519830 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.519793 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-credential-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.520050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.519859 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-token\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.520050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.519962 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n24vt\" (UniqueName: \"kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-kube-api-access-n24vt\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.520050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.520017 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-data\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.520160 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.520096 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.520160 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.520136 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-certs\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.520254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.520242 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8afe9b1-9212-4f51-874a-cb2520b26eae-istiod-ca-cert\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.520290 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.520274 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-envoy\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.520353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.520342 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-podinfo\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.621678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.621642 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-podinfo\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.621892 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.621699 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-credential-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.621892 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.621764 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-token\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.621892 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.621792 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n24vt\" (UniqueName: \"kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-kube-api-access-n24vt\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.621892 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.621825 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-data\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.621892 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.621856 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.622196 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.621911 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-certs\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.622196 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.622056 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8afe9b1-9212-4f51-874a-cb2520b26eae-istiod-ca-cert\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.622196 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.622103 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-envoy\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.622351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.622297 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-credential-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.622736 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.622710 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-data\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.622904 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.622879 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-certs\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.623191 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.623166 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.623636 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.623609 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8afe9b1-9212-4f51-874a-cb2520b26eae-istiod-ca-cert\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.624868 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.624841 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-podinfo\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.625109 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.625085 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-envoy\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.651317 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.651280 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-token\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.651484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.651393 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n24vt\" (UniqueName: \"kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-kube-api-access-n24vt\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.734686 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.734588 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw"] Oct 30 16:37:18.745497 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.745463 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.756714 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.756681 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw"] Oct 30 16:37:18.763346 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.763314 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2e0784\"/\"openshift-service-ca.crt\"" Oct 30 16:37:18.765320 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.765228 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2e0784\"/\"kube-root-ca.crt\"" Oct 30 16:37:18.765835 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.765808 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2e0784\"/\"distinct-2-w5-2e0784-kserve-self-signed-certs\"" Oct 30 16:37:18.765998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.765857 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2e0784\"/\"default-dockercfg-qzllj\"" Oct 30 16:37:18.780023 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.779989 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:18.805302 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.805263 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t"] Oct 30 16:37:18.811413 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.811369 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:18.825554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.824947 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-model-cache\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.825554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.825190 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-tmp-dir\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.825554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.825409 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-dshm\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.825554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.825500 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-home\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.825881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.825562 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzhhm\" (UniqueName: \"kubernetes.io/projected/da70d01c-6ab3-495f-8056-0546653f2a53-kube-api-access-kzhhm\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.825881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.825796 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da70d01c-6ab3-495f-8056-0546653f2a53-tls-certs\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.841760 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.841719 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t"] Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937132 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-data\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937221 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-tmp-dir\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937300 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istiod-ca-cert\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937343 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtzgz\" (UniqueName: \"kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-kube-api-access-xtzgz\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937372 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-podinfo\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937431 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-dshm\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937489 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-home\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937527 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-token\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937556 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzhhm\" (UniqueName: \"kubernetes.io/projected/da70d01c-6ab3-495f-8056-0546653f2a53-kube-api-access-kzhhm\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937590 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da70d01c-6ab3-495f-8056-0546653f2a53-tls-certs\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937614 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-envoy\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937715 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-model-cache\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937785 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-credential-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937814 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:18.986103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.937858 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-certs\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:18.991044 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.987608 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-tmp-dir\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.991044 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.988343 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-model-cache\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:18.991044 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:18.989871 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-home\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:19.023389 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.018569 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzhhm\" (UniqueName: \"kubernetes.io/projected/da70d01c-6ab3-495f-8056-0546653f2a53-kube-api-access-kzhhm\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.041426 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-certs\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.041592 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-data\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.041705 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istiod-ca-cert\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.041749 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzgz\" (UniqueName: \"kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-kube-api-access-xtzgz\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.041782 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-podinfo\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.041898 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-token\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.041945 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-envoy\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.042032 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-credential-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.042063 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.042169 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da70d01c-6ab3-495f-8056-0546653f2a53-tls-certs\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.042442 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.042707 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-certs\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.042940 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-data\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.043470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.043285 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-dshm\") pod \"distinct-2-w5-2e0784-kserve-755cbcf967-c99fw\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:19.044376 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.043563 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istiod-ca-cert\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.044376 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.043781 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-credential-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.054029 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.045382 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-podinfo\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.054029 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.047159 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-envoy\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.071004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.069605 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:19.083878 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.079688 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzgz\" (UniqueName: \"kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-kube-api-access-xtzgz\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.088152 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.088118 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-token\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.138125 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.138081 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:19.157745 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:19.157702 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8afe9b1_9212_4f51_874a_cb2520b26eae.slice/crio-f9ae516c29c478457b5c93ab66e04ce16c208bd6dcb5c98f42615625edfbdf7e WatchSource:0}: Error finding container f9ae516c29c478457b5c93ab66e04ce16c208bd6dcb5c98f42615625edfbdf7e: Status 404 returned error can't find the container with id f9ae516c29c478457b5c93ab66e04ce16c208bd6dcb5c98f42615625edfbdf7e Oct 30 16:37:19.161943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.161907 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:19.162059 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.162015 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:19.162126 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.162063 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:19.164125 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.162814 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg"] Oct 30 16:37:19.324363 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.324316 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4"] Oct 30 16:37:19.331176 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.331150 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.367012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.366533 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg"] Oct 30 16:37:19.377774 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.377676 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.378128 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.378106 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4"] Oct 30 16:37:19.395046 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.395007 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg"] Oct 30 16:37:19.410585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.410286 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw"] Oct 30 16:37:19.467594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.467553 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-envoy\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.468030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.467668 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-envoy\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.468030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.467712 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkphj\" (UniqueName: \"kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-kube-api-access-zkphj\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.468030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.467816 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.468030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.467855 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-data\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.468030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468016 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-podinfo\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.468312 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468088 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-certs\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.468312 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468253 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-token\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.468413 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468306 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-podinfo\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.468413 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468352 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istiod-ca-cert\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.468413 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468387 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv5bf\" (UniqueName: \"kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-kube-api-access-zv5bf\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.468557 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468464 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-credential-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.468610 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468569 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istiod-ca-cert\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.468663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468621 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.468711 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468662 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-credential-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.468763 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468705 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-data\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.468861 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468842 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-certs\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.468931 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.468875 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-token\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.490865 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.490798 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t"] Oct 30 16:37:19.492751 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:19.492723 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62ed47a_62cf_46a3_84f4_c06c2a13e22c.slice/crio-d3c8812ad29f56e132461018ac2e29b3b55bb1b624944ce5d416d272b71b5516 WatchSource:0}: Error finding container d3c8812ad29f56e132461018ac2e29b3b55bb1b624944ce5d416d272b71b5516: Status 404 returned error can't find the container with id d3c8812ad29f56e132461018ac2e29b3b55bb1b624944ce5d416d272b71b5516 Oct 30 16:37:19.496227 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.496188 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:19.496336 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.496267 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:19.496336 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.496310 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:19.570577 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.570418 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-credential-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.570577 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.570562 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istiod-ca-cert\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.570787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.570667 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.570787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.570699 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-credential-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.570787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.570736 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-data\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.570787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.570780 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-certs\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.571050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.570815 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-token\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.571050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.570839 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-envoy\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.571050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.570877 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-envoy\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.571050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.570903 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkphj\" (UniqueName: \"kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-kube-api-access-zkphj\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.571050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.570952 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.571050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.571002 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-data\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.571050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.571021 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-podinfo\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.571384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.571049 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-credential-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.571384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.571060 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-certs\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.571384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.571114 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.571384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.571217 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-token\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.571384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.571258 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-podinfo\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.571384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.571291 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istiod-ca-cert\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.571384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.571317 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv5bf\" (UniqueName: \"kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-kube-api-access-zv5bf\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.571384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.571354 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-certs\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.572200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.571654 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-data\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.572200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.571672 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-certs\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.572422 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.572399 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istiod-ca-cert\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.572492 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.572421 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istiod-ca-cert\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.572695 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.572669 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.573088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.573059 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-data\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.573515 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.573463 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-credential-socket\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.573811 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.573782 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-envoy\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.575312 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.575290 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-podinfo\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.575585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.575561 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-envoy\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.576187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.576166 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-podinfo\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.600666 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.600595 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-token\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.601024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.600858 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv5bf\" (UniqueName: \"kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-kube-api-access-zv5bf\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.601358 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.601316 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-token\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.601915 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.601866 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkphj\" (UniqueName: \"kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-kube-api-access-zkphj\") pod \"e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.673633 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.673596 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:19.700043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.700012 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:19.813067 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.811715 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" event={"ID":"e8afe9b1-9212-4f51-874a-cb2520b26eae","Type":"ContainerStarted","Data":"49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e"} Oct 30 16:37:19.813067 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.811828 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" event={"ID":"e8afe9b1-9212-4f51-874a-cb2520b26eae","Type":"ContainerStarted","Data":"f9ae516c29c478457b5c93ab66e04ce16c208bd6dcb5c98f42615625edfbdf7e"} Oct 30 16:37:19.825925 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.825848 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" event={"ID":"d62ed47a-62cf-46a3-84f4-c06c2a13e22c","Type":"ContainerStarted","Data":"c65f96cb2c08540b928d3fedbb077a96413a748c15e5ce1e1ca73c3fb9292794"} Oct 30 16:37:19.826215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.826177 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" event={"ID":"d62ed47a-62cf-46a3-84f4-c06c2a13e22c","Type":"ContainerStarted","Data":"d3c8812ad29f56e132461018ac2e29b3b55bb1b624944ce5d416d272b71b5516"} Oct 30 16:37:19.829926 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.829898 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" event={"ID":"da70d01c-6ab3-495f-8056-0546653f2a53","Type":"ContainerStarted","Data":"6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9"} Oct 30 16:37:19.830596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.830575 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" event={"ID":"da70d01c-6ab3-495f-8056-0546653f2a53","Type":"ContainerStarted","Data":"24f1d5505fa0301d746c51ee8afbc85c748f1fe8a33af578c8a6d642fbdf98d1"} Oct 30 16:37:19.865215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.865144 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" podStartSLOduration=1.865124147 podStartE2EDuration="1.865124147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:19.857202332 +0000 UTC m=+1001.605791744" watchObservedRunningTime="2026-10-30 16:37:19.865124147 +0000 UTC m=+1001.613713536" Oct 30 16:37:19.916107 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.916042 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" podStartSLOduration=1.916021806 podStartE2EDuration="1.916021806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:19.909502076 +0000 UTC m=+1001.658091504" watchObservedRunningTime="2026-10-30 16:37:19.916021806 +0000 UTC m=+1001.664611196" Oct 30 16:37:19.956063 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:19.955928 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podStartSLOduration=1.9559126 podStartE2EDuration="1.9559126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:19.95112248 +0000 UTC m=+1001.699711867" watchObservedRunningTime="2026-10-30 16:37:19.9559126 +0000 UTC m=+1001.704501986" Oct 30 16:37:20.034757 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:20.034721 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792c09a1_9d64_470a_8bd8_b2e1cef34b7a.slice/crio-7cd8654f5e9b0a1d483f6c5f1fb27e6156b68e2a5a865f8cf71b8610ffcf9b8f WatchSource:0}: Error finding container 7cd8654f5e9b0a1d483f6c5f1fb27e6156b68e2a5a865f8cf71b8610ffcf9b8f: Status 404 returned error can't find the container with id 7cd8654f5e9b0a1d483f6c5f1fb27e6156b68e2a5a865f8cf71b8610ffcf9b8f Oct 30 16:37:20.036052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.036014 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4"] Oct 30 16:37:20.043643 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.042134 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:20.043643 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.042228 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:20.043643 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.042271 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:20.056705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.056678 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg"] Oct 30 16:37:20.058030 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:20.058001 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6b5808_eb84_4f36_8d25_0c5b19ab3c46.slice/crio-8eb37cc693329394ae8d9a6a5e2bb152abdd6f274c9a96be4a514fafa9b642e5 WatchSource:0}: Error finding container 8eb37cc693329394ae8d9a6a5e2bb152abdd6f274c9a96be4a514fafa9b642e5: Status 404 returned error can't find the container with id 8eb37cc693329394ae8d9a6a5e2bb152abdd6f274c9a96be4a514fafa9b642e5 Oct 30 16:37:20.061020 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.060936 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:20.061239 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.061211 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:20.061391 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.061369 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:20.071813 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.070697 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:20.138473 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.138443 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:20.156738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.156693 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:37:20.157036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.156996 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 16:37:20.781185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.781147 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:20.784213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.784182 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:37:20.784369 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.784241 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:37:20.883065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.883026 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" event={"ID":"792c09a1-9d64-470a-8bd8-b2e1cef34b7a","Type":"ContainerStarted","Data":"f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a"} Oct 30 16:37:20.883347 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.883319 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" event={"ID":"792c09a1-9d64-470a-8bd8-b2e1cef34b7a","Type":"ContainerStarted","Data":"7cd8654f5e9b0a1d483f6c5f1fb27e6156b68e2a5a865f8cf71b8610ffcf9b8f"} Oct 30 16:37:20.886574 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.886056 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" event={"ID":"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46","Type":"ContainerStarted","Data":"e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced"} Oct 30 16:37:20.886574 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:20.886234 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" event={"ID":"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46","Type":"ContainerStarted","Data":"8eb37cc693329394ae8d9a6a5e2bb152abdd6f274c9a96be4a514fafa9b642e5"} Oct 30 16:37:21.058093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:21.057947 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" podStartSLOduration=2.057926645 podStartE2EDuration="2.057926645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:20.954625063 +0000 UTC m=+1002.703214465" watchObservedRunningTime="2026-10-30 16:37:21.057926645 +0000 UTC m=+1002.806516034" Oct 30 16:37:21.087329 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:21.087256 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" podStartSLOduration=2.087235062 podStartE2EDuration="2.087235062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:21.014618892 +0000 UTC m=+1002.763208304" watchObservedRunningTime="2026-10-30 16:37:21.087235062 +0000 UTC m=+1002.835824449" Oct 30 16:37:21.143410 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:21.143372 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:37:21.143647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:21.143621 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 16:37:21.677004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:21.676746 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:21.680998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:21.679581 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:37:21.680998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:21.679652 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 16:37:21.702748 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:21.702714 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:21.704762 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:21.704734 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:37:21.705022 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:21.704997 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 16:37:21.787142 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:21.787099 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:37:21.788837 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:21.788800 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:37:22.146991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.146438 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:37:22.146991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.146506 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 16:37:22.351545 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.351492 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:37:22.425054 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.416582 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:22.425054 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.418038 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:22.472530 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.472063 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:37:22.681051 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.680485 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:37:22.681051 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.680563 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 16:37:22.704457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.704418 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:37:22.704707 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.704678 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 16:37:22.704943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.704908 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:22.783439 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.783396 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:37:22.783748 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.783716 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:37:22.836816 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.836773 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:22.949456 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:22.949350 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:23.010628 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.010588 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:23.186992 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.184326 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:37:23.186992 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.184825 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 16:37:23.442601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.442450 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:23.447102 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.447017 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:37:23.532995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.507546 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:37:23.533704 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.533669 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:37:23.598994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.559686 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/health\": context deadline exceeded" Oct 30 16:37:23.687997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.621108 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:23.687997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.621631 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:37:23.853591 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.781218 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:37:23.853591 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.789136 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:23.853591 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.848440 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:23.877215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.874815 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:37:23.877215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.876117 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:23.892126 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.884407 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:37:23.939287 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.937242 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:37:23.939287 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:23.937381 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:37:24.072026 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.052701 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:37:24.072026 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.054507 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 16:37:24.135738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.135628 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 16:37:24.136064 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.136026 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 16:37:24.180999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.153115 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:37:24.180999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.156579 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 16:37:24.463166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.463061 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 16:37:24.638116 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.638062 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:24.778001 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.777031 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:37:24.778001 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.777110 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 16:37:24.785009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.778660 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:37:24.785009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.778716 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 16:37:24.848998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.848929 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:37:24.901691 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.896784 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:37:24.901691 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.898154 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:37:24.901691 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.899260 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 16:37:24.907671 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.907618 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:37:24.907922 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.907894 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:37:24.992209 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.992169 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:37:24.992493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:24.992464 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 16:37:25.184227 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.184127 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:37:25.220239 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.220193 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 16:37:25.275171 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.269299 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:25.392944 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.392900 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:25.484065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.481772 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:25.663135 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.663074 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:25.675361 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.675319 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:37:25.675609 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.675577 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 16:37:25.706008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.705321 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:37:25.706008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.705378 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 16:37:25.789996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.789883 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:37:25.789996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.789944 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:37:25.851626 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.851574 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:25.916080 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.914101 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 16:37:25.916080 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.914582 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:37:25.916615 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:25.916500 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 16:37:26.126066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.105800 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 16:37:26.151996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.139513 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:37:26.151996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.148257 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 16:37:26.279416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.279372 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:26.486059 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.482055 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:26.486059 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.485931 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:26.691033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.665171 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:26.691033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.684919 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Oct 30 16:37:26.691033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.685329 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:37:26.691033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.685378 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 16:37:26.716395 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.716208 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:37:26.716395 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.716296 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 16:37:26.803056 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.800659 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:37:26.803056 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.800727 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:37:26.914904 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.914860 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:26.919285 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:26.919240 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:27.006016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.005524 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:37:27.102771 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.102670 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:37:27.105197 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.105171 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:37:27.179129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.178632 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:37:27.179129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.178688 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 16:37:27.208631 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.208601 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:37:27.312050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.312020 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:37:27.774095 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.774058 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:37:27.778176 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.777269 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 16:37:27.778176 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.775897 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:37:27.778176 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.777663 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 16:37:27.899104 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.899072 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:27.899937 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.899869 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:27.903122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:27.903085 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:37:28.656840 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:28.656807 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:28.659374 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:28.659350 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:28.673631 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:28.673355 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:37:28.674295 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:28.674263 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:37:28.674501 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:28.674478 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 16:37:28.701329 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:28.701292 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:37:28.701472 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:28.701358 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 16:37:29.075819 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:29.075780 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:29.100817 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:29.100772 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:29.678835 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:29.678803 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:29.679334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:29.679231 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:29.679955 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:29.679934 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:37:29.920428 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:29.920387 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:29.922304 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:29.922270 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:29.925632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:29.923792 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:37:30.082314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:30.082284 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:37:31.797846 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:37:31.797804 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:37:34.250189 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.250150 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw"] Oct 30 16:37:34.255325 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.255297 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.283215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.283177 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-podinfo\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.283422 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.283236 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istiod-ca-cert\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.283491 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.283419 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.283491 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.283448 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-certs\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.283491 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.283478 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-envoy\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.283648 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.283549 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-data\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.283794 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.283771 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz9cl\" (UniqueName: \"kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-kube-api-access-hz9cl\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.283901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.283886 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-credential-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.283957 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.283915 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-token\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.289585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.289551 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw"] Oct 30 16:37:34.385205 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.385166 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-podinfo\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.385383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.385213 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istiod-ca-cert\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.385383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.385297 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.385383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.385314 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-certs\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.385383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.385334 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-envoy\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.385383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.385363 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-data\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.385661 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.385421 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz9cl\" (UniqueName: \"kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-kube-api-access-hz9cl\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.385661 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.385502 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-credential-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.385661 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.385526 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-token\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.385881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.385855 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-certs\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.386009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.385954 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-data\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.386129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.386074 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-credential-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.386205 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.386144 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-socket\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.386205 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.386196 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istiod-ca-cert\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.387692 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.387672 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-envoy\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.387839 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.387822 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-podinfo\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.408312 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.408282 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-token\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.408612 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.408587 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz9cl\" (UniqueName: \"kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-kube-api-access-hz9cl\") pod \"e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.572087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.571994 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:34.959526 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.959391 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw"] Oct 30 16:37:34.962408 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:34.962375 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2bb0a9_1ccd_4187_bb7a_ffe7871527bf.slice/crio-c531686b06b4e62c5547593e52c8ad6254f3573c3519fe7b402057a8c34d9aac WatchSource:0}: Error finding container c531686b06b4e62c5547593e52c8ad6254f3573c3519fe7b402057a8c34d9aac: Status 404 returned error can't find the container with id c531686b06b4e62c5547593e52c8ad6254f3573c3519fe7b402057a8c34d9aac Oct 30 16:37:34.965640 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.965053 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:34.965640 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.965125 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:34.965640 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:34.965154 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:37:35.161999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:35.160554 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" event={"ID":"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf","Type":"ContainerStarted","Data":"84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213"} Oct 30 16:37:35.161999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:35.160678 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" event={"ID":"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf","Type":"ContainerStarted","Data":"c531686b06b4e62c5547593e52c8ad6254f3573c3519fe7b402057a8c34d9aac"} Oct 30 16:37:35.204545 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:35.204473 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" podStartSLOduration=1.204456695 podStartE2EDuration="1.204456695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:35.203519758 +0000 UTC m=+1016.952109197" watchObservedRunningTime="2026-10-30 16:37:35.204456695 +0000 UTC m=+1016.953046084" Oct 30 16:37:35.572761 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:35.572723 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:36.574254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:36.574218 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:37:36.575020 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:36.574951 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 16:37:37.240064 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.240025 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp"] Oct 30 16:37:37.247145 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.247110 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp"] Oct 30 16:37:37.247327 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.247164 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:37.255897 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.255751 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d64ct\" (UniqueName: \"kubernetes.io/projected/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-kube-api-access-d64ct\") pod \"maas-api-revoke-sub-9bbe1b9f-xcvlp\" (UID: \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:37.255897 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.255814 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bbe1b9f-xcvlp\" (UID: \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:37.256140 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.256116 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bbe1b9f-xcvlp\" (UID: \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:37.262192 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.262166 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:37:37.273648 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.273623 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:37:37.357158 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.357121 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bbe1b9f-xcvlp\" (UID: \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:37.357348 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.357217 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bbe1b9f-xcvlp\" (UID: \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:37.357400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.357365 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d64ct\" (UniqueName: \"kubernetes.io/projected/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-kube-api-access-d64ct\") pod \"maas-api-revoke-sub-9bbe1b9f-xcvlp\" (UID: \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:37.357806 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.357784 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bbe1b9f-xcvlp\" (UID: \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:37.359530 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.359509 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bbe1b9f-xcvlp\" (UID: \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:37.381170 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.381134 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d64ct\" (UniqueName: \"kubernetes.io/projected/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-kube-api-access-d64ct\") pod \"maas-api-revoke-sub-9bbe1b9f-xcvlp\" (UID: \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:37.568047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.567347 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:37.579017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.578985 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:37:37.579465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.579432 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 16:37:37.901630 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:37.901592 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp"] Oct 30 16:37:37.903336 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:37.903296 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c612c8_2f17_4a5e_88f5_4927cd45e4cd.slice/crio-14579642b0ddcd145f29933537e735cddd0d5d4b3e4c45896890401265b626be WatchSource:0}: Error finding container 14579642b0ddcd145f29933537e735cddd0d5d4b3e4c45896890401265b626be: Status 404 returned error can't find the container with id 14579642b0ddcd145f29933537e735cddd0d5d4b3e4c45896890401265b626be Oct 30 16:37:38.245319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:38.245285 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" event={"ID":"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd","Type":"ContainerStarted","Data":"14579642b0ddcd145f29933537e735cddd0d5d4b3e4c45896890401265b626be"} Oct 30 16:37:38.573709 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:38.573634 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:37:38.573709 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:38.573688 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 16:37:39.281010 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:39.280934 5838 generic.go:356] "Generic (PLEG): container finished" podID="b9c612c8-2f17-4a5e-88f5-4927cd45e4cd" containerID="91e701d09ad297cf7a5566d60e6d8ec252939b1fba5283c3df317ced0ddaab8f" exitCode=0 Oct 30 16:37:39.282413 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:39.281928 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" event={"ID":"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd","Type":"ContainerDied","Data":"91e701d09ad297cf7a5566d60e6d8ec252939b1fba5283c3df317ced0ddaab8f"} Oct 30 16:37:39.561633 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:39.561597 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:37:39.573121 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:39.573086 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:37:39.573305 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:39.573142 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 16:37:40.509566 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:40.509536 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:40.572654 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:40.572618 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:37:40.572852 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:40.572670 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 16:37:40.654681 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:40.654632 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-openshift-ca\") pod \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\" (UID: \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\") " Oct 30 16:37:40.654859 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:40.654788 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-xks-ca\") pod \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\" (UID: \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\") " Oct 30 16:37:40.654931 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:40.654894 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-kube-api-access-d64ct\" (UniqueName: \"kubernetes.io/projected/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-kube-api-access-d64ct\") pod \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\" (UID: \"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd\") " Oct 30 16:37:40.655362 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:40.655335 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-openshift-ca" pod "b9c612c8-2f17-4a5e-88f5-4927cd45e4cd" (UID: "b9c612c8-2f17-4a5e-88f5-4927cd45e4cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:40.655655 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:40.655635 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:40.658229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:40.658195 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-kube-api-access-d64ct" pod "b9c612c8-2f17-4a5e-88f5-4927cd45e4cd" (UID: "b9c612c8-2f17-4a5e-88f5-4927cd45e4cd"). InnerVolumeSpecName "kube-api-access-d64ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:40.659264 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:40.659234 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-xks-ca" pod "b9c612c8-2f17-4a5e-88f5-4927cd45e4cd" (UID: "b9c612c8-2f17-4a5e-88f5-4927cd45e4cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:40.759995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:40.759271 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:40.759995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:40.759313 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d64ct\" (UniqueName: \"kubernetes.io/projected/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd-kube-api-access-d64ct\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:41.339177 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:41.327062 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" event={"ID":"b9c612c8-2f17-4a5e-88f5-4927cd45e4cd","Type":"ContainerDied","Data":"14579642b0ddcd145f29933537e735cddd0d5d4b3e4c45896890401265b626be"} Oct 30 16:37:41.339177 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:41.327137 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14579642b0ddcd145f29933537e735cddd0d5d4b3e4c45896890401265b626be" Oct 30 16:37:41.339177 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:41.327214 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp" Oct 30 16:37:41.585997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:41.574497 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:37:41.585997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:41.574560 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 16:37:42.357995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:42.349466 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:42.529266 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:42.529216 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 16:37:42.575996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:42.575523 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:37:42.575996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:42.575604 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 16:37:42.713654 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:42.713549 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:42.715831 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:42.714392 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 16:37:42.852107 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:42.852056 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:42.947482 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:42.944068 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:42.947482 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:42.946380 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 16:37:42.995118 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:42.995070 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:43.081569 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.080926 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:43.290184 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.290077 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:43.349671 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.349631 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:37:43.362268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.362239 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:37:43.564083 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.538152 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:43.564083 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.538495 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:43.564083 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.539773 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:37:43.628997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.625742 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 16:37:43.667086 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.663060 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 16:37:43.735153 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.716398 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 16:37:43.865053 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.859536 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:43.865053 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.861629 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:37:43.865053 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.861693 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:37:43.911998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.890887 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:43.912869 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.912841 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:43.945199 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.945158 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:44.012213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.957952 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:37:44.012213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.969017 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 16:37:44.012213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.969038 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 16:37:44.012213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:43.969413 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:37:44.392353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:44.367124 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 16:37:44.392353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:44.369932 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:37:44.392353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:44.370852 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:37:44.541011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:44.525451 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:44.541011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:44.525523 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:44.550005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:44.548327 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:44.916003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:44.877062 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 16:37:45.064026 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:44.950603 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:45.104041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.064567 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:45.104041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.065809 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:37:45.104041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.066607 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:37:45.104041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.100108 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": context deadline exceeded" Oct 30 16:37:45.491995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.483355 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 16:37:45.491995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.483746 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 30 16:37:45.491995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.483795 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:37:45.491995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.484661 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" podUID="9f6b4090-8bc3-4396-9f43-9150857119f8" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.17:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:45.501847 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:37:45.501807 5838 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.107s" Oct 30 16:37:45.554040 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.553999 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:45.585151 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.585108 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:45.596001 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.585416 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:45.779052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.738088 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:45.779052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.738162 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:45.940141 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.902699 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:46.002049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:45.976058 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:37:46.018867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.018184 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:46.018867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.018258 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:46.092190 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.090958 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:46.092190 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.091082 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:46.092190 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.091386 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 16:37:46.092190 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.091621 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:37:46.150443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.123360 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:46.150443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.125023 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:46.150443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.146610 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:37:46.244156 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.205751 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:37:46.244156 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.241986 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:37:46.274996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.246786 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:37:46.408730 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.408623 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:37:46.453022 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.437782 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:37:46.598349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.598058 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:46.626286 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.626254 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:37:46.752352 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.722418 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:37:46.778998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.767434 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:37:46.860067 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.859314 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s"] Oct 30 16:37:46.860410 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.860363 5838 cpu_manager.go:414] "containerMap: removing container" podUID="b9c612c8-2f17-4a5e-88f5-4927cd45e4cd" containerName="revoke-keys" Oct 30 16:37:46.860410 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.860389 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c612c8-2f17-4a5e-88f5-4927cd45e4cd" containerName="revoke-keys" Oct 30 16:37:46.860581 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.860547 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c612c8-2f17-4a5e-88f5-4927cd45e4cd" containerName="revoke-keys" Oct 30 16:37:46.867215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.867188 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:46.873267 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.873236 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s"] Oct 30 16:37:46.930483 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.930453 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ccf5c9\"/\"distinct-2-w3-ccf5c9-kserve-self-signed-certs\"" Oct 30 16:37:46.956600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.956559 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp862\" (UniqueName: \"kubernetes.io/projected/281c4576-fb69-4e08-bae2-dc19a6b93eae-kube-api-access-qp862\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:46.956600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.956614 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-home\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:46.957110 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.956795 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-dshm\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:46.957110 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.956926 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-tmp-dir\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:46.957110 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.957055 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-model-cache\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:46.957301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.957277 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/281c4576-fb69-4e08-bae2-dc19a6b93eae-tls-certs\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:46.968717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.968635 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct"] Oct 30 16:37:46.974677 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.974646 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:46.999636 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:46.999599 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct"] Oct 30 16:37:47.022040 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.021937 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:37:47.022182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.022129 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:37:47.059021 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.058963 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7skd\" (UniqueName: \"kubernetes.io/projected/18151a38-458f-4a95-a49f-19ee96bf9c52-kube-api-access-z7skd\") pod \"maas-api-revoke-sub-f7a065f3-xcxct\" (UID: \"18151a38-458f-4a95-a49f-19ee96bf9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:47.059375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.059117 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/281c4576-fb69-4e08-bae2-dc19a6b93eae-tls-certs\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.059375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.059163 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a065f3-xcxct\" (UID: \"18151a38-458f-4a95-a49f-19ee96bf9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:47.059375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.059230 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp862\" (UniqueName: \"kubernetes.io/projected/281c4576-fb69-4e08-bae2-dc19a6b93eae-kube-api-access-qp862\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.059375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.059257 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-home\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.059375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.059319 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a065f3-xcxct\" (UID: \"18151a38-458f-4a95-a49f-19ee96bf9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:47.059375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.059359 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-dshm\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.059744 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.059428 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-tmp-dir\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.059744 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.059518 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-model-cache\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.061017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.060029 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-home\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.061017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.060116 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-tmp-dir\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.061017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.060465 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-model-cache\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.062213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.062167 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-dshm\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.062344 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.062322 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/281c4576-fb69-4e08-bae2-dc19a6b93eae-tls-certs\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.083477 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.083449 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp862\" (UniqueName: \"kubernetes.io/projected/281c4576-fb69-4e08-bae2-dc19a6b93eae-kube-api-access-qp862\") pod \"distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.161360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.161323 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7skd\" (UniqueName: \"kubernetes.io/projected/18151a38-458f-4a95-a49f-19ee96bf9c52-kube-api-access-z7skd\") pod \"maas-api-revoke-sub-f7a065f3-xcxct\" (UID: \"18151a38-458f-4a95-a49f-19ee96bf9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:47.161560 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.161390 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a065f3-xcxct\" (UID: \"18151a38-458f-4a95-a49f-19ee96bf9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:47.161560 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.161461 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a065f3-xcxct\" (UID: \"18151a38-458f-4a95-a49f-19ee96bf9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:47.162356 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.162324 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a065f3-xcxct\" (UID: \"18151a38-458f-4a95-a49f-19ee96bf9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:47.166756 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.166728 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a065f3-xcxct\" (UID: \"18151a38-458f-4a95-a49f-19ee96bf9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:47.195533 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.195492 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:47.228493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.225695 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7skd\" (UniqueName: \"kubernetes.io/projected/18151a38-458f-4a95-a49f-19ee96bf9c52-kube-api-access-z7skd\") pod \"maas-api-revoke-sub-f7a065f3-xcxct\" (UID: \"18151a38-458f-4a95-a49f-19ee96bf9c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:47.300663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.300328 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:47.753894 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.753862 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct"] Oct 30 16:37:47.755020 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:47.754959 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18151a38_458f_4a95_a49f_19ee96bf9c52.slice/crio-1706ec1e34adfc3b70b65e61dcad8f888f222e2bd7ab052fdd85b22b7fedb06d WatchSource:0}: Error finding container 1706ec1e34adfc3b70b65e61dcad8f888f222e2bd7ab052fdd85b22b7fedb06d: Status 404 returned error can't find the container with id 1706ec1e34adfc3b70b65e61dcad8f888f222e2bd7ab052fdd85b22b7fedb06d Oct 30 16:37:47.759632 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:47.759603 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod281c4576_fb69_4e08_bae2_dc19a6b93eae.slice/crio-770b582759d62a14974cdf8aeffba466f5313f0bbd9a67d13229c7e6cd953105 WatchSource:0}: Error finding container 770b582759d62a14974cdf8aeffba466f5313f0bbd9a67d13229c7e6cd953105: Status 404 returned error can't find the container with id 770b582759d62a14974cdf8aeffba466f5313f0bbd9a67d13229c7e6cd953105 Oct 30 16:37:47.760047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:47.759926 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s"] Oct 30 16:37:48.155692 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:48.155655 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" event={"ID":"281c4576-fb69-4e08-bae2-dc19a6b93eae","Type":"ContainerStarted","Data":"7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1"} Oct 30 16:37:48.156103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:48.155699 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" event={"ID":"281c4576-fb69-4e08-bae2-dc19a6b93eae","Type":"ContainerStarted","Data":"770b582759d62a14974cdf8aeffba466f5313f0bbd9a67d13229c7e6cd953105"} Oct 30 16:37:48.158206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:48.158131 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" event={"ID":"18151a38-458f-4a95-a49f-19ee96bf9c52","Type":"ContainerStarted","Data":"f5e16d138ad281ea82f1a7b986fe90079605e318d460a45487caf9716b9a0991"} Oct 30 16:37:48.158360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:48.158239 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" event={"ID":"18151a38-458f-4a95-a49f-19ee96bf9c52","Type":"ContainerStarted","Data":"1706ec1e34adfc3b70b65e61dcad8f888f222e2bd7ab052fdd85b22b7fedb06d"} Oct 30 16:37:48.196077 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:48.196046 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:48.208693 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:48.208640 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podStartSLOduration=2.208626819 podStartE2EDuration="2.208626819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:48.202820119 +0000 UTC m=+1029.951409506" watchObservedRunningTime="2026-10-30 16:37:48.208626819 +0000 UTC m=+1029.957216244" Oct 30 16:37:48.244762 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:48.244708 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" podStartSLOduration=2.244692503 podStartE2EDuration="2.244692503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:48.24289547 +0000 UTC m=+1029.991484877" watchObservedRunningTime="2026-10-30 16:37:48.244692503 +0000 UTC m=+1029.993281890" Oct 30 16:37:49.168228 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:49.168194 5838 generic.go:356] "Generic (PLEG): container finished" podID="18151a38-458f-4a95-a49f-19ee96bf9c52" containerID="f5e16d138ad281ea82f1a7b986fe90079605e318d460a45487caf9716b9a0991" exitCode=0 Oct 30 16:37:49.168623 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:49.168289 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" event={"ID":"18151a38-458f-4a95-a49f-19ee96bf9c52","Type":"ContainerDied","Data":"f5e16d138ad281ea82f1a7b986fe90079605e318d460a45487caf9716b9a0991"} Oct 30 16:37:50.342020 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:50.341983 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:50.444108 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:50.417557 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-openshift-ca\") pod \"18151a38-458f-4a95-a49f-19ee96bf9c52\" (UID: \"18151a38-458f-4a95-a49f-19ee96bf9c52\") " Oct 30 16:37:50.444108 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:50.417762 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18151a38-458f-4a95-a49f-19ee96bf9c52-kube-api-access-z7skd\" (UniqueName: \"kubernetes.io/projected/18151a38-458f-4a95-a49f-19ee96bf9c52-kube-api-access-z7skd\") pod \"18151a38-458f-4a95-a49f-19ee96bf9c52\" (UID: \"18151a38-458f-4a95-a49f-19ee96bf9c52\") " Oct 30 16:37:50.444108 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:50.417857 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-xks-ca\") pod \"18151a38-458f-4a95-a49f-19ee96bf9c52\" (UID: \"18151a38-458f-4a95-a49f-19ee96bf9c52\") " Oct 30 16:37:50.444108 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:50.435491 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-xks-ca" pod "18151a38-458f-4a95-a49f-19ee96bf9c52" (UID: "18151a38-458f-4a95-a49f-19ee96bf9c52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:50.444108 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:50.435874 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-openshift-ca" pod "18151a38-458f-4a95-a49f-19ee96bf9c52" (UID: "18151a38-458f-4a95-a49f-19ee96bf9c52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:50.475056 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:50.449234 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18151a38-458f-4a95-a49f-19ee96bf9c52-kube-api-access-z7skd" pod "18151a38-458f-4a95-a49f-19ee96bf9c52" (UID: "18151a38-458f-4a95-a49f-19ee96bf9c52"). InnerVolumeSpecName "kube-api-access-z7skd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:50.522057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:50.520790 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7skd\" (UniqueName: \"kubernetes.io/projected/18151a38-458f-4a95-a49f-19ee96bf9c52-kube-api-access-z7skd\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:50.522057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:50.520827 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:50.522057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:50.520841 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18151a38-458f-4a95-a49f-19ee96bf9c52-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:51.082133 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.082093 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2"] Oct 30 16:37:51.083101 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.083079 5838 cpu_manager.go:414] "containerMap: removing container" podUID="18151a38-458f-4a95-a49f-19ee96bf9c52" containerName="revoke-keys" Oct 30 16:37:51.083101 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.083100 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18151a38-458f-4a95-a49f-19ee96bf9c52" containerName="revoke-keys" Oct 30 16:37:51.083247 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.083219 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="18151a38-458f-4a95-a49f-19ee96bf9c52" containerName="revoke-keys" Oct 30 16:37:51.087412 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.087387 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:51.102299 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.102269 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2"] Oct 30 16:37:51.127889 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.127851 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4697e804-bb04-4e91-a607-3be930573237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57857170-t48k2\" (UID: \"4697e804-bb04-4e91-a607-3be930573237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:51.128121 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.128078 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4697e804-bb04-4e91-a607-3be930573237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57857170-t48k2\" (UID: \"4697e804-bb04-4e91-a607-3be930573237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:51.128239 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.128220 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf2q9\" (UniqueName: \"kubernetes.io/projected/4697e804-bb04-4e91-a607-3be930573237-kube-api-access-wf2q9\") pod \"maas-api-revoke-sub-57857170-t48k2\" (UID: \"4697e804-bb04-4e91-a607-3be930573237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:51.191009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.190961 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" Oct 30 16:37:51.191200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.190960 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct" event={"ID":"18151a38-458f-4a95-a49f-19ee96bf9c52","Type":"ContainerDied","Data":"1706ec1e34adfc3b70b65e61dcad8f888f222e2bd7ab052fdd85b22b7fedb06d"} Oct 30 16:37:51.191200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.191102 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1706ec1e34adfc3b70b65e61dcad8f888f222e2bd7ab052fdd85b22b7fedb06d" Oct 30 16:37:51.229071 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.229033 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4697e804-bb04-4e91-a607-3be930573237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57857170-t48k2\" (UID: \"4697e804-bb04-4e91-a607-3be930573237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:51.229273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.229145 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4697e804-bb04-4e91-a607-3be930573237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57857170-t48k2\" (UID: \"4697e804-bb04-4e91-a607-3be930573237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:51.229273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.229212 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf2q9\" (UniqueName: \"kubernetes.io/projected/4697e804-bb04-4e91-a607-3be930573237-kube-api-access-wf2q9\") pod \"maas-api-revoke-sub-57857170-t48k2\" (UID: \"4697e804-bb04-4e91-a607-3be930573237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:51.229832 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.229810 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4697e804-bb04-4e91-a607-3be930573237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57857170-t48k2\" (UID: \"4697e804-bb04-4e91-a607-3be930573237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:51.231452 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.231428 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4697e804-bb04-4e91-a607-3be930573237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57857170-t48k2\" (UID: \"4697e804-bb04-4e91-a607-3be930573237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:51.252542 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.252509 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf2q9\" (UniqueName: \"kubernetes.io/projected/4697e804-bb04-4e91-a607-3be930573237-kube-api-access-wf2q9\") pod \"maas-api-revoke-sub-57857170-t48k2\" (UID: \"4697e804-bb04-4e91-a607-3be930573237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:51.408012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.407879 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:51.730156 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:51.730122 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2"] Oct 30 16:37:51.731224 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:51.731191 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4697e804_bb04_4e91_a607_3be930573237.slice/crio-c7eab0f90b2fe69fa8057c6ed05d02fb95a3e074c807f2a646dde8c35c2acc3a WatchSource:0}: Error finding container c7eab0f90b2fe69fa8057c6ed05d02fb95a3e074c807f2a646dde8c35c2acc3a: Status 404 returned error can't find the container with id c7eab0f90b2fe69fa8057c6ed05d02fb95a3e074c807f2a646dde8c35c2acc3a Oct 30 16:37:52.205433 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:52.205393 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" event={"ID":"4697e804-bb04-4e91-a607-3be930573237","Type":"ContainerStarted","Data":"03bdba7b14e3781d8fd49d9cd6858c3121292ed7d4d6f4fe4049e8ae8a4dda7b"} Oct 30 16:37:52.205627 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:52.205460 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" event={"ID":"4697e804-bb04-4e91-a607-3be930573237","Type":"ContainerStarted","Data":"c7eab0f90b2fe69fa8057c6ed05d02fb95a3e074c807f2a646dde8c35c2acc3a"} Oct 30 16:37:52.247111 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:52.247054 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" podStartSLOduration=1.2470394200000001 podStartE2EDuration="1.24703942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:52.245854264 +0000 UTC m=+1033.994443686" watchObservedRunningTime="2026-10-30 16:37:52.24703942 +0000 UTC m=+1033.995628806" Oct 30 16:37:53.005334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.005286 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr"] Oct 30 16:37:53.010548 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.010520 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:53.031826 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.031783 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr"] Oct 30 16:37:53.055883 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.055845 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr\" (UID: \"f3e048cd-1dbc-4926-b951-2d5719660c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:53.056154 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.056136 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr\" (UID: \"f3e048cd-1dbc-4926-b951-2d5719660c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:53.056217 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.056179 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxt6f\" (UniqueName: \"kubernetes.io/projected/f3e048cd-1dbc-4926-b951-2d5719660c41-kube-api-access-fxt6f\") pod \"maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr\" (UID: \"f3e048cd-1dbc-4926-b951-2d5719660c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:53.158001 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.157928 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr\" (UID: \"f3e048cd-1dbc-4926-b951-2d5719660c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:53.158257 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.158017 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxt6f\" (UniqueName: \"kubernetes.io/projected/f3e048cd-1dbc-4926-b951-2d5719660c41-kube-api-access-fxt6f\") pod \"maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr\" (UID: \"f3e048cd-1dbc-4926-b951-2d5719660c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:53.158257 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.158091 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr\" (UID: \"f3e048cd-1dbc-4926-b951-2d5719660c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:53.158821 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.158801 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr\" (UID: \"f3e048cd-1dbc-4926-b951-2d5719660c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:53.160475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.160446 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr\" (UID: \"f3e048cd-1dbc-4926-b951-2d5719660c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:53.181080 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.181041 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxt6f\" (UniqueName: \"kubernetes.io/projected/f3e048cd-1dbc-4926-b951-2d5719660c41-kube-api-access-fxt6f\") pod \"maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr\" (UID: \"f3e048cd-1dbc-4926-b951-2d5719660c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:53.216032 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.215952 5838 generic.go:356] "Generic (PLEG): container finished" podID="4697e804-bb04-4e91-a607-3be930573237" containerID="03bdba7b14e3781d8fd49d9cd6858c3121292ed7d4d6f4fe4049e8ae8a4dda7b" exitCode=0 Oct 30 16:37:53.216032 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.216009 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" event={"ID":"4697e804-bb04-4e91-a607-3be930573237","Type":"ContainerDied","Data":"03bdba7b14e3781d8fd49d9cd6858c3121292ed7d4d6f4fe4049e8ae8a4dda7b"} Oct 30 16:37:53.332416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.332324 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:53.652475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:53.652444 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr"] Oct 30 16:37:53.653853 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:37:53.653829 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e048cd_1dbc_4926_b951_2d5719660c41.slice/crio-ad1e113a1004b6afaa27e1b37f9a5a4c2331f0c4e2c37612078c1fe806bdbe4d WatchSource:0}: Error finding container ad1e113a1004b6afaa27e1b37f9a5a4c2331f0c4e2c37612078c1fe806bdbe4d: Status 404 returned error can't find the container with id ad1e113a1004b6afaa27e1b37f9a5a4c2331f0c4e2c37612078c1fe806bdbe4d Oct 30 16:37:54.227353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.227251 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" event={"ID":"f3e048cd-1dbc-4926-b951-2d5719660c41","Type":"ContainerStarted","Data":"d0a0139b58a1c802770a12bf2c5c89a6e014c0d01f9df0d5129e5f3f3d771de6"} Oct 30 16:37:54.227353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.227312 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" event={"ID":"f3e048cd-1dbc-4926-b951-2d5719660c41","Type":"ContainerStarted","Data":"ad1e113a1004b6afaa27e1b37f9a5a4c2331f0c4e2c37612078c1fe806bdbe4d"} Oct 30 16:37:54.267896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.267845 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" podStartSLOduration=2.267830808 podStartE2EDuration="2.267830808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:54.267028918 +0000 UTC m=+1036.015618325" watchObservedRunningTime="2026-10-30 16:37:54.267830808 +0000 UTC m=+1036.016420191" Oct 30 16:37:54.422351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.422322 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:54.504185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.504153 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4697e804-bb04-4e91-a607-3be930573237-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4697e804-bb04-4e91-a607-3be930573237-maas-api-xks-ca\") pod \"4697e804-bb04-4e91-a607-3be930573237\" (UID: \"4697e804-bb04-4e91-a607-3be930573237\") " Oct 30 16:37:54.504417 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.504256 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4697e804-bb04-4e91-a607-3be930573237-kube-api-access-wf2q9\" (UniqueName: \"kubernetes.io/projected/4697e804-bb04-4e91-a607-3be930573237-kube-api-access-wf2q9\") pod \"4697e804-bb04-4e91-a607-3be930573237\" (UID: \"4697e804-bb04-4e91-a607-3be930573237\") " Oct 30 16:37:54.504495 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.504433 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4697e804-bb04-4e91-a607-3be930573237-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4697e804-bb04-4e91-a607-3be930573237-maas-api-openshift-ca\") pod \"4697e804-bb04-4e91-a607-3be930573237\" (UID: \"4697e804-bb04-4e91-a607-3be930573237\") " Oct 30 16:37:54.504857 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.504823 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4697e804-bb04-4e91-a607-3be930573237-maas-api-openshift-ca" pod "4697e804-bb04-4e91-a607-3be930573237" (UID: "4697e804-bb04-4e91-a607-3be930573237"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:54.505434 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.505412 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4697e804-bb04-4e91-a607-3be930573237-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:54.506557 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.506529 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4697e804-bb04-4e91-a607-3be930573237-kube-api-access-wf2q9" pod "4697e804-bb04-4e91-a607-3be930573237" (UID: "4697e804-bb04-4e91-a607-3be930573237"). InnerVolumeSpecName "kube-api-access-wf2q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:54.506886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.506865 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4697e804-bb04-4e91-a607-3be930573237-maas-api-xks-ca" pod "4697e804-bb04-4e91-a607-3be930573237" (UID: "4697e804-bb04-4e91-a607-3be930573237"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:54.607225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.607191 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4697e804-bb04-4e91-a607-3be930573237-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:54.607225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:54.607219 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf2q9\" (UniqueName: \"kubernetes.io/projected/4697e804-bb04-4e91-a607-3be930573237-kube-api-access-wf2q9\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:55.238507 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:55.238472 5838 generic.go:356] "Generic (PLEG): container finished" podID="f3e048cd-1dbc-4926-b951-2d5719660c41" containerID="d0a0139b58a1c802770a12bf2c5c89a6e014c0d01f9df0d5129e5f3f3d771de6" exitCode=0 Oct 30 16:37:55.239053 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:55.238580 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" event={"ID":"f3e048cd-1dbc-4926-b951-2d5719660c41","Type":"ContainerDied","Data":"d0a0139b58a1c802770a12bf2c5c89a6e014c0d01f9df0d5129e5f3f3d771de6"} Oct 30 16:37:55.240679 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:55.240653 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" event={"ID":"4697e804-bb04-4e91-a607-3be930573237","Type":"ContainerDied","Data":"c7eab0f90b2fe69fa8057c6ed05d02fb95a3e074c807f2a646dde8c35c2acc3a"} Oct 30 16:37:55.240814 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:55.240691 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7eab0f90b2fe69fa8057c6ed05d02fb95a3e074c807f2a646dde8c35c2acc3a" Oct 30 16:37:55.240814 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:55.240746 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2" Oct 30 16:37:56.446430 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:56.446400 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:56.540193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:56.540097 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-openshift-ca\") pod \"f3e048cd-1dbc-4926-b951-2d5719660c41\" (UID: \"f3e048cd-1dbc-4926-b951-2d5719660c41\") " Oct 30 16:37:56.540345 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:56.540304 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e048cd-1dbc-4926-b951-2d5719660c41-kube-api-access-fxt6f\" (UniqueName: \"kubernetes.io/projected/f3e048cd-1dbc-4926-b951-2d5719660c41-kube-api-access-fxt6f\") pod \"f3e048cd-1dbc-4926-b951-2d5719660c41\" (UID: \"f3e048cd-1dbc-4926-b951-2d5719660c41\") " Oct 30 16:37:56.540484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:56.540469 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-xks-ca\") pod \"f3e048cd-1dbc-4926-b951-2d5719660c41\" (UID: \"f3e048cd-1dbc-4926-b951-2d5719660c41\") " Oct 30 16:37:56.540605 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:56.540565 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-openshift-ca" pod "f3e048cd-1dbc-4926-b951-2d5719660c41" (UID: "f3e048cd-1dbc-4926-b951-2d5719660c41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:56.541427 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:56.541398 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:56.542945 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:56.542911 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-xks-ca" pod "f3e048cd-1dbc-4926-b951-2d5719660c41" (UID: "f3e048cd-1dbc-4926-b951-2d5719660c41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:56.543109 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:56.543090 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e048cd-1dbc-4926-b951-2d5719660c41-kube-api-access-fxt6f" pod "f3e048cd-1dbc-4926-b951-2d5719660c41" (UID: "f3e048cd-1dbc-4926-b951-2d5719660c41"). InnerVolumeSpecName "kube-api-access-fxt6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:56.642374 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:56.642332 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxt6f\" (UniqueName: \"kubernetes.io/projected/f3e048cd-1dbc-4926-b951-2d5719660c41-kube-api-access-fxt6f\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:56.642374 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:56.642367 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e048cd-1dbc-4926-b951-2d5719660c41-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:37:57.196082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:57.196038 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:57.209464 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:57.209431 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:37:57.261453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:57.261422 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" Oct 30 16:37:57.261650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:57.261465 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr" event={"ID":"f3e048cd-1dbc-4926-b951-2d5719660c41","Type":"ContainerDied","Data":"ad1e113a1004b6afaa27e1b37f9a5a4c2331f0c4e2c37612078c1fe806bdbe4d"} Oct 30 16:37:57.261650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:57.261522 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad1e113a1004b6afaa27e1b37f9a5a4c2331f0c4e2c37612078c1fe806bdbe4d" Oct 30 16:37:57.273513 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:37:57.273474 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:38:00.962020 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:00.957068 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:38:01.068063 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:01.067081 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:01.068063 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:01.067513 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 16:38:01.087996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:01.084136 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:01.102994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:01.098073 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:01.271995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:01.271387 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:01.383279 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:01.383135 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:01.499883 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:01.499837 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:01.557518 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:01.557416 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 16:38:01.886134 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:01.886075 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:01.970011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:01.969101 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:01.970011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:01.969916 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:38:02.010745 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:02.010699 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:38:02.077023 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.070091 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 16:38:02.077023 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.070500 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:02.077023 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.071189 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:38:02.077023 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.071253 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:38:02.096011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.094257 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:02.096011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.095488 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:38:02.169053 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.168922 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:38:02.192656 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.169567 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:38:02.273170 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.273102 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:02.273990 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.273921 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:38:02.414124 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.414077 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:02.564834 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.526328 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:02.592960 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.583068 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 16:38:02.897092 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.889306 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 16:38:03.012067 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:02.972892 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:03.041449 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.041396 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded" Oct 30 16:38:03.222062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.132294 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 16:38:03.249995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.132340 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:03.249995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.132803 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": context deadline exceeded" Oct 30 16:38:03.249995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.187066 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:38:03.682998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.669126 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 16:38:03.683341 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:03.683277 5838 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.288s" Oct 30 16:38:03.683485 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.683468 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:38:03.707995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.688158 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:38:03.707995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.694426 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 16:38:04.012065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.871139 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:04.012065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.879241 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:04.012065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.879905 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" podUID="9f6b4090-8bc3-4396-9f43-9150857119f8" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.17:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:04.012065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.890280 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:38:04.012065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.897172 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/health\": context deadline exceeded" Oct 30 16:38:04.012065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.959395 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:04.012065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.959462 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:04.012065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:03.985542 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:04.049041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:04.035184 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:04.126089 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:04.068539 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 16:38:04.235996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:04.210499 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:38:04.317238 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:04.317121 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:04.398008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:04.222787 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:04.520294 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:04.520145 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:04.721057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:04.709205 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:04.721057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:04.709574 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 16:38:04.734784 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:04.733410 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:38:04.890140 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:04.876285 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:04.915150 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:04.914195 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:05.175998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.172463 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw"] Oct 30 16:38:05.175998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.174687 5838 cpu_manager.go:414] "containerMap: removing container" podUID="4697e804-bb04-4e91-a607-3be930573237" containerName="revoke-keys" Oct 30 16:38:05.175998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.174726 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4697e804-bb04-4e91-a607-3be930573237" containerName="revoke-keys" Oct 30 16:38:05.175998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.174932 5838 cpu_manager.go:414] "containerMap: removing container" podUID="f3e048cd-1dbc-4926-b951-2d5719660c41" containerName="revoke-keys" Oct 30 16:38:05.175998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.174943 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e048cd-1dbc-4926-b951-2d5719660c41" containerName="revoke-keys" Oct 30 16:38:05.175998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.175400 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e048cd-1dbc-4926-b951-2d5719660c41" containerName="revoke-keys" Oct 30 16:38:05.175998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.175425 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="4697e804-bb04-4e91-a607-3be930573237" containerName="revoke-keys" Oct 30 16:38:05.271009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.231880 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:05.361002 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.341078 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:05.362033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.361960 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:05.410223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.407576 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:05.482055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.481068 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:05.527152 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.527092 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:05.692059 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.692023 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv"] Oct 30 16:38:05.697461 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.692377 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:05.710839 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.710807 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw"] Oct 30 16:38:05.724524 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.711021 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:05.731999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.711204 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:05.731999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.711223 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:05.744077 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.737276 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:38:05.776390 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.776336 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" podUID="9f6b4090-8bc3-4396-9f43-9150857119f8" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.17:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:05.814751 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.814721 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv"] Oct 30 16:38:05.999202 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:05.999089 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:06.094095 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.028446 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 16:38:06.137418 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.074588 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.137721 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.137690 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkq2s\" (UniqueName: \"kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-kube-api-access-rkq2s\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.137874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.137857 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-credential-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322186 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-token\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322273 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-podinfo\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322304 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322373 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-data\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322420 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-envoy\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322452 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clzqg\" (UniqueName: \"kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-kube-api-access-clzqg\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322588 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-token\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322650 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-certs\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322719 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istiod-ca-cert\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322754 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-credential-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322821 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-envoy\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322872 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-data\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.322918 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-certs\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.323010 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istiod-ca-cert\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.323037 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-podinfo\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.330055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.286289 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz"] Oct 30 16:38:06.399002 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.374963 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:38:06.399002 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.375461 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.416822 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428016 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkq2s\" (UniqueName: \"kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-kube-api-access-rkq2s\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428068 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-credential-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428154 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-token\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428194 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-podinfo\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428220 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428270 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-data\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428310 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-envoy\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428339 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzqg\" (UniqueName: \"kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-kube-api-access-clzqg\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428451 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-token\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428500 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-certs\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428553 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istiod-ca-cert\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428587 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-credential-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428645 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-envoy\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428683 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-data\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428712 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-certs\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428771 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istiod-ca-cert\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428797 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-podinfo\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.428869 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.429393 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.444006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.429470 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 16:38:06.484166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.447881 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-certs\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.484166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.470411 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-credential-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.484166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.470866 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-data\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.484166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.471100 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.484166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.471295 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-credential-socket\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.484166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.472141 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istiod-ca-cert\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.484166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.472730 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istiod-ca-cert\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.484166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.479297 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-data\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.484166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.480050 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-certs\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.567523 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.536700 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:06.776995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.738754 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:38:06.797993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.757154 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 16:38:06.820278 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.758489 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-envoy\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.820889 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.820755 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20e8844-c4fd-40fb-884e-5c53aa339890-istiod-ca-cert\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.823720 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.823669 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.823939 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.823915 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-certs\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.824176 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.824156 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-podinfo\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.824529 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.824493 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-data\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.824807 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.824787 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgjbv\" (UniqueName: \"kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-kube-api-access-lgjbv\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.824997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.824965 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-token\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.825245 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.825226 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-credential-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.826641 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.759798 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 16:38:06.828139 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.793026 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.922921 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.922885 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzqg\" (UniqueName: \"kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-kube-api-access-clzqg\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.953637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.953601 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Oct 30 16:38:06.955741 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.955488 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-data\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.955741 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.955591 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgjbv\" (UniqueName: \"kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-kube-api-access-lgjbv\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.955741 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.955631 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-token\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.955741 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.955736 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-credential-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.956594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.955894 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-envoy\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.956594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.955983 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20e8844-c4fd-40fb-884e-5c53aa339890-istiod-ca-cert\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.956594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.956043 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.956594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.956074 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-certs\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.956594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.956103 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-podinfo\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.959364 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.959341 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-podinfo\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.961149 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.960371 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-data\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.961454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.961270 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-credential-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.963930 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.963818 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-envoy\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.965079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.964491 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20e8844-c4fd-40fb-884e-5c53aa339890-istiod-ca-cert\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.965079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.964721 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.965079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.964934 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-certs\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.966586 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.965882 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-token\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.966586 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.966137 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-envoy\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.967253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.966873 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-podinfo\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.976157 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.967874 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-podinfo\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.976157 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.968491 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkq2s\" (UniqueName: \"kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-kube-api-access-rkq2s\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.976157 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.971814 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-envoy\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:06.979363 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.979257 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-token\") pod \"e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:06.999471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.999440 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-token\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:06.999667 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.999589 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:38:06.999808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.999797 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:38:06.999914 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:06.999905 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:38:07.000043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.000033 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:38:07.000161 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.000152 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:38:07.000261 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.000251 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:38:07.000351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.000341 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz"] Oct 30 16:38:07.000463 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.000454 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:38:07.000565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.000557 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:38:07.000673 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.000665 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:38:07.001396 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.001341 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgjbv\" (UniqueName: \"kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-kube-api-access-lgjbv\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:07.002540 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.002514 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:07.003349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.003324 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:38:07.023989 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.017488 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:07.030610 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.030363 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz"] Oct 30 16:38:07.037919 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.037895 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj"] Oct 30 16:38:07.192492 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.192457 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:07.387239 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:38:07.387206 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ec9020_bd36_43a5_8b7a_8f753f7230b5.slice/crio-f218dbb22c95c2c570d92ec2b7d2c61a893e48834df732f6229156222e7b490e WatchSource:0}: Error finding container f218dbb22c95c2c570d92ec2b7d2c61a893e48834df732f6229156222e7b490e: Status 404 returned error can't find the container with id f218dbb22c95c2c570d92ec2b7d2c61a893e48834df732f6229156222e7b490e Oct 30 16:38:07.387688 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.387366 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw"] Oct 30 16:38:07.389650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.389617 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:07.389757 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.389700 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:07.389757 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.389736 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:07.404871 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.404836 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz"] Oct 30 16:38:07.420583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.420259 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:07.420583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.420342 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:07.420583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.420386 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:07.555682 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:38:07.555631 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3985ca_cecb_4c1a_9584_af2320d1c71e.slice/crio-143c3f2fd3e8452929a8bfeeb02f10bccbb76f457c43f667dfca28d7b77545d9 WatchSource:0}: Error finding container 143c3f2fd3e8452929a8bfeeb02f10bccbb76f457c43f667dfca28d7b77545d9: Status 404 returned error can't find the container with id 143c3f2fd3e8452929a8bfeeb02f10bccbb76f457c43f667dfca28d7b77545d9 Oct 30 16:38:07.559094 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.558614 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:07.559094 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.558711 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:07.559094 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.558758 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:07.560031 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.559534 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv"] Oct 30 16:38:07.938045 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:07.901729 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5"] Oct 30 16:38:08.047552 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.047519 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5"] Oct 30 16:38:08.047890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.047869 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:08.160314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.135333 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:38:08.160314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.135407 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:08.160314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.145554 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkxs8\" (UniqueName: \"kubernetes.io/projected/97b0a141-4f14-4f9a-8d19-7267172c99ae-kube-api-access-pkxs8\") pod \"maas-api-revoke-sub-0075b4bf-hczr5\" (UID: \"97b0a141-4f14-4f9a-8d19-7267172c99ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:08.160314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.145658 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0075b4bf-hczr5\" (UID: \"97b0a141-4f14-4f9a-8d19-7267172c99ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:08.160314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.145716 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0075b4bf-hczr5\" (UID: \"97b0a141-4f14-4f9a-8d19-7267172c99ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:08.225638 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.225506 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" event={"ID":"f3ec9020-bd36-43a5-8b7a-8f753f7230b5","Type":"ContainerStarted","Data":"f1d54b292dc3ce76921d02b6ceb56166d19df6edf83a0de6d0483ddc40e0845c"} Oct 30 16:38:08.229139 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.229102 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" event={"ID":"f3ec9020-bd36-43a5-8b7a-8f753f7230b5","Type":"ContainerStarted","Data":"f218dbb22c95c2c570d92ec2b7d2c61a893e48834df732f6229156222e7b490e"} Oct 30 16:38:08.254944 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.254901 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" event={"ID":"e20e8844-c4fd-40fb-884e-5c53aa339890","Type":"ContainerStarted","Data":"0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267"} Oct 30 16:38:08.257125 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.257095 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" event={"ID":"e20e8844-c4fd-40fb-884e-5c53aa339890","Type":"ContainerStarted","Data":"c093eb777661cc64ec49b73a73bf9e943b874a3a373ef29c6a725b1bb0267ad6"} Oct 30 16:38:08.257327 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.257311 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz"] Oct 30 16:38:08.268045 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.255923 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkxs8\" (UniqueName: \"kubernetes.io/projected/97b0a141-4f14-4f9a-8d19-7267172c99ae-kube-api-access-pkxs8\") pod \"maas-api-revoke-sub-0075b4bf-hczr5\" (UID: \"97b0a141-4f14-4f9a-8d19-7267172c99ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:08.268203 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.268103 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0075b4bf-hczr5\" (UID: \"97b0a141-4f14-4f9a-8d19-7267172c99ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:08.268203 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.268176 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0075b4bf-hczr5\" (UID: \"97b0a141-4f14-4f9a-8d19-7267172c99ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:08.269405 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.269379 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0075b4bf-hczr5\" (UID: \"97b0a141-4f14-4f9a-8d19-7267172c99ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:08.272281 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.272208 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0075b4bf-hczr5\" (UID: \"97b0a141-4f14-4f9a-8d19-7267172c99ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:08.276693 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.276670 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:08.282807 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.282778 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkxs8\" (UniqueName: \"kubernetes.io/projected/97b0a141-4f14-4f9a-8d19-7267172c99ae-kube-api-access-pkxs8\") pod \"maas-api-revoke-sub-0075b4bf-hczr5\" (UID: \"97b0a141-4f14-4f9a-8d19-7267172c99ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:08.283665 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.283623 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" event={"ID":"5a3985ca-cecb-4c1a-9584-af2320d1c71e","Type":"ContainerStarted","Data":"a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0"} Oct 30 16:38:08.283889 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.283854 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" event={"ID":"5a3985ca-cecb-4c1a-9584-af2320d1c71e","Type":"ContainerStarted","Data":"143c3f2fd3e8452929a8bfeeb02f10bccbb76f457c43f667dfca28d7b77545d9"} Oct 30 16:38:08.292877 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.292817 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz"] Oct 30 16:38:08.305105 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.297622 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" podStartSLOduration=5.297603029 podStartE2EDuration="5.297603029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:08.294739279 +0000 UTC m=+1050.043328690" watchObservedRunningTime="2026-10-30 16:38:08.297603029 +0000 UTC m=+1050.046192417" Oct 30 16:38:08.369927 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.369882 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz\" (UID: \"7acae368-f382-4336-b418-221bd12ebe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:08.370171 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.370003 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6spw9\" (UniqueName: \"kubernetes.io/projected/7acae368-f382-4336-b418-221bd12ebe8e-kube-api-access-6spw9\") pod \"maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz\" (UID: \"7acae368-f382-4336-b418-221bd12ebe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:08.370171 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.370090 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz\" (UID: \"7acae368-f382-4336-b418-221bd12ebe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:08.379403 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.379324 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" podStartSLOduration=5.37930477 podStartE2EDuration="5.37930477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:08.3767658 +0000 UTC m=+1050.125355214" watchObservedRunningTime="2026-10-30 16:38:08.37930477 +0000 UTC m=+1050.127894158" Oct 30 16:38:08.423890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.423833 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" podStartSLOduration=4.423817096 podStartE2EDuration="4.423817096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:08.41936017 +0000 UTC m=+1050.167949582" watchObservedRunningTime="2026-10-30 16:38:08.423817096 +0000 UTC m=+1050.172406482" Oct 30 16:38:08.472284 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.472248 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz\" (UID: \"7acae368-f382-4336-b418-221bd12ebe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:08.472505 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.472316 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6spw9\" (UniqueName: \"kubernetes.io/projected/7acae368-f382-4336-b418-221bd12ebe8e-kube-api-access-6spw9\") pod \"maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz\" (UID: \"7acae368-f382-4336-b418-221bd12ebe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:08.472505 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.472368 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz\" (UID: \"7acae368-f382-4336-b418-221bd12ebe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:08.473109 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.473084 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz\" (UID: \"7acae368-f382-4336-b418-221bd12ebe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:08.474723 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.474700 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz\" (UID: \"7acae368-f382-4336-b418-221bd12ebe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:08.476817 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.476771 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:08.498061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.498027 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6spw9\" (UniqueName: \"kubernetes.io/projected/7acae368-f382-4336-b418-221bd12ebe8e-kube-api-access-6spw9\") pod \"maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz\" (UID: \"7acae368-f382-4336-b418-221bd12ebe8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:08.639955 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.636460 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:08.874206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:08.874096 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5"] Oct 30 16:38:09.025059 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:09.021342 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:09.025059 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:09.023873 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.025059 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:09.023934 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 16:38:09.032999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:09.028248 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:09.090013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:09.088109 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.090013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:09.088166 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" podUID="e20e8844-c4fd-40fb-884e-5c53aa339890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:38:09.238010 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:09.224392 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:09.389816 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:09.297826 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz"] Oct 30 16:38:09.480998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:09.479017 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.480998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:09.479091 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:38:09.506805 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:09.506770 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" event={"ID":"97b0a141-4f14-4f9a-8d19-7267172c99ae","Type":"ContainerStarted","Data":"5622aa0d41688b4006d25c8e05c1ced0fef9e16b57b26330c903fdd4a37efccc"} Oct 30 16:38:10.006893 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:10.004839 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 16:38:10.006893 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:10.004890 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 16:38:10.036599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:10.036543 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:38:10.036770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:10.036636 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" podUID="e20e8844-c4fd-40fb-884e-5c53aa339890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:38:10.194265 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:10.193371 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:38:10.194265 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:10.193423 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:38:10.590995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:10.574666 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" event={"ID":"97b0a141-4f14-4f9a-8d19-7267172c99ae","Type":"ContainerStarted","Data":"94cf546ae3a4251684c08ae17af80bcb0171d18b84e8fa73508b986529721389"} Oct 30 16:38:10.593085 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:10.593036 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" event={"ID":"7acae368-f382-4336-b418-221bd12ebe8e","Type":"ContainerStarted","Data":"f32b928015922398650bb14eb66ca09ed2e8baf0bb16f40363e3b2324fe8b8aa"} Oct 30 16:38:10.593249 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:10.593109 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" event={"ID":"7acae368-f382-4336-b418-221bd12ebe8e","Type":"ContainerStarted","Data":"6b199ff9448668cf7956111efab951e2ec185108340fee95cb9e4cff44e099d5"} Oct 30 16:38:10.689992 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:10.689141 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" podStartSLOduration=3.689120333 podStartE2EDuration="3.689120333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:10.668748176 +0000 UTC m=+1052.417337579" watchObservedRunningTime="2026-10-30 16:38:10.689120333 +0000 UTC m=+1052.437709724" Oct 30 16:38:10.748783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:10.739812 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" podStartSLOduration=2.739789749 podStartE2EDuration="2.739789749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:10.713257786 +0000 UTC m=+1052.461847185" watchObservedRunningTime="2026-10-30 16:38:10.739789749 +0000 UTC m=+1052.488379139" Oct 30 16:38:11.004510 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:11.004476 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 16:38:11.004648 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:11.004540 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 16:38:11.023855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:11.023748 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:38:11.023855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:11.023802 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" podUID="e20e8844-c4fd-40fb-884e-5c53aa339890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:38:11.198225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:11.195392 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:38:11.198225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:11.195453 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:38:11.605000 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:11.604937 5838 generic.go:356] "Generic (PLEG): container finished" podID="97b0a141-4f14-4f9a-8d19-7267172c99ae" containerID="94cf546ae3a4251684c08ae17af80bcb0171d18b84e8fa73508b986529721389" exitCode=0 Oct 30 16:38:11.605422 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:11.605025 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" event={"ID":"97b0a141-4f14-4f9a-8d19-7267172c99ae","Type":"ContainerDied","Data":"94cf546ae3a4251684c08ae17af80bcb0171d18b84e8fa73508b986529721389"} Oct 30 16:38:11.606733 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:11.606706 5838 generic.go:356] "Generic (PLEG): container finished" podID="7acae368-f382-4336-b418-221bd12ebe8e" containerID="f32b928015922398650bb14eb66ca09ed2e8baf0bb16f40363e3b2324fe8b8aa" exitCode=0 Oct 30 16:38:11.606866 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:11.606771 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" event={"ID":"7acae368-f382-4336-b418-221bd12ebe8e","Type":"ContainerDied","Data":"f32b928015922398650bb14eb66ca09ed2e8baf0bb16f40363e3b2324fe8b8aa"} Oct 30 16:38:12.004389 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:12.004358 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 16:38:12.004594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:12.004574 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 16:38:12.035996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:12.034921 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" podUID="2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3" containerName="payload-processing" containerID="cri-o://d4590db00d1a7ffe97be06b277da0c9db96b480e55b4a81b844e295353efc916" gracePeriod=25 Oct 30 16:38:12.036838 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:12.036809 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:38:12.037034 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:12.037008 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" podUID="e20e8844-c4fd-40fb-884e-5c53aa339890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:38:12.060111 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:12.039020 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" podUID="02bc187b-d110-4d74-b2b9-2b96919c1a2a" containerName="payload-pre-processing" containerID="cri-o://b49659f70c245c3be13625c92279396aebe364884c4e4ee8905a335d9839237b" gracePeriod=25 Oct 30 16:38:12.193025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:12.192951 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:38:12.193292 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:12.193261 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:38:13.008031 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:13.001338 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn"] Oct 30 16:38:13.105749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:13.105712 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:38:13.117998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:13.106426 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" podUID="e20e8844-c4fd-40fb-884e-5c53aa339890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:38:13.117998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:13.105847 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 16:38:13.117998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:13.106521 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 16:38:13.249958 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:13.249913 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:38:13.250321 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:13.250258 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:38:13.502078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:13.466100 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg"] Oct 30 16:38:14.010690 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.006808 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:14.010690 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.006811 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:14.010690 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.008015 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 16:38:14.010690 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.008069 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 16:38:14.170060 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.169777 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:14.191008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.179219 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 16:38:14.191008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.183182 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:38:14.191008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.183229 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" podUID="e20e8844-c4fd-40fb-884e-5c53aa339890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:38:14.201010 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.195250 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:38:14.201010 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.195312 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:38:14.206268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.206243 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" event={"ID":"7acae368-f382-4336-b418-221bd12ebe8e","Type":"ContainerDied","Data":"6b199ff9448668cf7956111efab951e2ec185108340fee95cb9e4cff44e099d5"} Oct 30 16:38:14.206406 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.206320 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b199ff9448668cf7956111efab951e2ec185108340fee95cb9e4cff44e099d5" Oct 30 16:38:14.206932 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.206911 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:14.209411 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.209393 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:14.297011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.296501 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-openshift-ca\") pod \"97b0a141-4f14-4f9a-8d19-7267172c99ae\" (UID: \"97b0a141-4f14-4f9a-8d19-7267172c99ae\") " Oct 30 16:38:14.297011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.296747 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-openshift-ca\") pod \"7acae368-f382-4336-b418-221bd12ebe8e\" (UID: \"7acae368-f382-4336-b418-221bd12ebe8e\") " Oct 30 16:38:14.297011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.296814 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b0a141-4f14-4f9a-8d19-7267172c99ae-kube-api-access-pkxs8\" (UniqueName: \"kubernetes.io/projected/97b0a141-4f14-4f9a-8d19-7267172c99ae-kube-api-access-pkxs8\") pod \"97b0a141-4f14-4f9a-8d19-7267172c99ae\" (UID: \"97b0a141-4f14-4f9a-8d19-7267172c99ae\") " Oct 30 16:38:14.297011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.296844 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-xks-ca\") pod \"7acae368-f382-4336-b418-221bd12ebe8e\" (UID: \"7acae368-f382-4336-b418-221bd12ebe8e\") " Oct 30 16:38:14.297011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.296922 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-xks-ca\") pod \"97b0a141-4f14-4f9a-8d19-7267172c99ae\" (UID: \"97b0a141-4f14-4f9a-8d19-7267172c99ae\") " Oct 30 16:38:14.297011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.296963 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7acae368-f382-4336-b418-221bd12ebe8e-kube-api-access-6spw9\" (UniqueName: \"kubernetes.io/projected/7acae368-f382-4336-b418-221bd12ebe8e-kube-api-access-6spw9\") pod \"7acae368-f382-4336-b418-221bd12ebe8e\" (UID: \"7acae368-f382-4336-b418-221bd12ebe8e\") " Oct 30 16:38:14.298696 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.298354 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-openshift-ca" pod "97b0a141-4f14-4f9a-8d19-7267172c99ae" (UID: "97b0a141-4f14-4f9a-8d19-7267172c99ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:14.298696 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.298653 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-openshift-ca" pod "7acae368-f382-4336-b418-221bd12ebe8e" (UID: "7acae368-f382-4336-b418-221bd12ebe8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:14.299960 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.299932 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b0a141-4f14-4f9a-8d19-7267172c99ae-kube-api-access-pkxs8" pod "97b0a141-4f14-4f9a-8d19-7267172c99ae" (UID: "97b0a141-4f14-4f9a-8d19-7267172c99ae"). InnerVolumeSpecName "kube-api-access-pkxs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:14.300582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.300554 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-xks-ca" pod "97b0a141-4f14-4f9a-8d19-7267172c99ae" (UID: "97b0a141-4f14-4f9a-8d19-7267172c99ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:14.302423 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.302401 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7acae368-f382-4336-b418-221bd12ebe8e-kube-api-access-6spw9" pod "7acae368-f382-4336-b418-221bd12ebe8e" (UID: "7acae368-f382-4336-b418-221bd12ebe8e"). InnerVolumeSpecName "kube-api-access-6spw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:14.304455 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.304434 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-xks-ca" pod "7acae368-f382-4336-b418-221bd12ebe8e" (UID: "7acae368-f382-4336-b418-221bd12ebe8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:14.405071 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.405024 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:14.405282 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.405267 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkxs8\" (UniqueName: \"kubernetes.io/projected/97b0a141-4f14-4f9a-8d19-7267172c99ae-kube-api-access-pkxs8\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:14.405362 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.405351 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7acae368-f382-4336-b418-221bd12ebe8e-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:14.405471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.405459 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:14.405655 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.405642 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6spw9\" (UniqueName: \"kubernetes.io/projected/7acae368-f382-4336-b418-221bd12ebe8e-kube-api-access-6spw9\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:14.405825 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:14.405813 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b0a141-4f14-4f9a-8d19-7267172c99ae-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:15.004330 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:15.004284 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 16:38:15.004494 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:15.004352 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 16:38:15.023752 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:15.023712 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:38:15.024171 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:15.023773 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" podUID="e20e8844-c4fd-40fb-884e-5c53aa339890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:38:15.193655 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:15.193616 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:38:15.193854 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:15.193687 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:38:15.250551 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:15.250516 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz" Oct 30 16:38:15.252694 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:15.252067 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" Oct 30 16:38:15.256950 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:15.252916 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5" event={"ID":"97b0a141-4f14-4f9a-8d19-7267172c99ae","Type":"ContainerDied","Data":"5622aa0d41688b4006d25c8e05c1ced0fef9e16b57b26330c903fdd4a37efccc"} Oct 30 16:38:15.256950 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:15.252987 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5622aa0d41688b4006d25c8e05c1ced0fef9e16b57b26330c903fdd4a37efccc" Oct 30 16:38:16.030660 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:16.030622 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:16.031213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:16.030914 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:16.031858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:16.031835 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:38:16.421441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:16.421331 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:16.423014 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:16.422963 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:16.427858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:16.427823 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:38:16.557589 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:16.557522 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:16.558558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:16.558530 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:16.559568 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:16.559545 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:38:18.588682 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.588644 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv"] Oct 30 16:38:18.637798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.634312 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj"] Oct 30 16:38:18.637798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.635490 5838 cpu_manager.go:414] "containerMap: removing container" podUID="97b0a141-4f14-4f9a-8d19-7267172c99ae" containerName="revoke-keys" Oct 30 16:38:18.637798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.635511 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b0a141-4f14-4f9a-8d19-7267172c99ae" containerName="revoke-keys" Oct 30 16:38:18.637798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.635527 5838 cpu_manager.go:414] "containerMap: removing container" podUID="7acae368-f382-4336-b418-221bd12ebe8e" containerName="revoke-keys" Oct 30 16:38:18.637798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.635535 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7acae368-f382-4336-b418-221bd12ebe8e" containerName="revoke-keys" Oct 30 16:38:18.637798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.635689 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b0a141-4f14-4f9a-8d19-7267172c99ae" containerName="revoke-keys" Oct 30 16:38:18.637798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.635703 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="7acae368-f382-4336-b418-221bd12ebe8e" containerName="revoke-keys" Oct 30 16:38:18.643568 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.643130 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.670307 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.670269 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj"] Oct 30 16:38:18.766784 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.766741 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-certs\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.766996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.766813 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istiod-ca-cert\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.766996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.766959 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkjmr\" (UniqueName: \"kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-kube-api-access-nkjmr\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.767108 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.767077 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-data\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.767150 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.767119 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-credential-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.767270 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.767248 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-token\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.767335 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.767298 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.767443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.767427 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-envoy\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.767496 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.767482 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-podinfo\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.868477 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.868383 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-podinfo\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.868673 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.868524 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-certs\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.868673 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.868579 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istiod-ca-cert\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.868673 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.868625 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkjmr\" (UniqueName: \"kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-kube-api-access-nkjmr\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.868843 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.868677 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-data\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.868843 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.868708 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-credential-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.868843 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.868768 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-token\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.868843 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.868801 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.869166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.868854 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-envoy\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.869166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.869044 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-certs\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.869166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.869112 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-data\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.869274 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.869249 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-credential-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.869274 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.869261 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-socket\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.869503 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.869482 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istiod-ca-cert\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.871174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.871150 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-envoy\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.871276 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.871227 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-podinfo\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.891739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.891696 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkjmr\" (UniqueName: \"kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-kube-api-access-nkjmr\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.892160 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.892133 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-token\") pod \"e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:18.998689 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:18.998652 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:19.034995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.025983 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf"] Oct 30 16:38:19.063282 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.063252 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.092570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.087896 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-credential-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.092570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.088000 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-token\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.092570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.088043 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-podinfo\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.092570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.088104 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istiod-ca-cert\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.092570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.088138 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-envoy\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.092570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.088243 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-data\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.092570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.088312 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.092570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.088352 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqfls\" (UniqueName: \"kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-kube-api-access-hqfls\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.092570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.088378 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-certs\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.094607 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.094209 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf"] Oct 30 16:38:19.189717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.189679 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqfls\" (UniqueName: \"kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-kube-api-access-hqfls\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.189925 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.189725 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-certs\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.189925 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.189859 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-credential-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.190052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.189942 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-token\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.190052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.190024 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-podinfo\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.190177 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.190125 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istiod-ca-cert\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.190231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.190184 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-envoy\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.191076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.190291 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-credential-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.191076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.190340 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-data\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.191076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.190287 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-certs\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.191076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.190396 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.191076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.190603 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-socket\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.191076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.190760 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-data\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.191076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.191023 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istiod-ca-cert\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.193860 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.193813 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-podinfo\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.194314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.194290 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-envoy\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.309985 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.309927 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqfls\" (UniqueName: \"kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-kube-api-access-hqfls\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.391851 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.373609 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-token\") pod \"e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.476996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.473209 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:19.743319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.743272 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj"] Oct 30 16:38:19.747594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.747536 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:19.747732 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.747673 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:19.747732 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.747724 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:19.909502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.909469 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" podUID="2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:38:19.909502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:19.909493 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" podUID="02bc187b-d110-4d74-b2b9-2b96919c1a2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:38:20.041392 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.041361 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf"] Oct 30 16:38:20.042373 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:38:20.042346 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13833b5b_ab9b_4a32_811b_5eb07645cfa9.slice/crio-b096f40c06100a9bed18f65c3231846bc8b2f56c85fe41e769d91365e6f8352f WatchSource:0}: Error finding container b096f40c06100a9bed18f65c3231846bc8b2f56c85fe41e769d91365e6f8352f: Status 404 returned error can't find the container with id b096f40c06100a9bed18f65c3231846bc8b2f56c85fe41e769d91365e6f8352f Oct 30 16:38:20.045384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.045347 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:20.045466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.045436 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:20.045508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.045484 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:38:20.373984 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.373892 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h"] Oct 30 16:38:20.379222 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.379195 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:20.399286 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.399256 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:20.401286 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.401259 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:38:20.404846 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.404817 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h"] Oct 30 16:38:20.407444 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.407421 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd"] Oct 30 16:38:20.461569 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.461517 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:20.477073 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.477047 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd"] Oct 30 16:38:20.478777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.478746 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aeddee8-bsg9h\" (UID: \"a2bfb675-2581-4e1b-a305-215a157b86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:20.478877 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.478817 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl2cn\" (UniqueName: \"kubernetes.io/projected/a2bfb675-2581-4e1b-a305-215a157b86dd-kube-api-access-tl2cn\") pod \"maas-api-revoke-sub-2aeddee8-bsg9h\" (UID: \"a2bfb675-2581-4e1b-a305-215a157b86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:20.479237 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.479211 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aeddee8-bsg9h\" (UID: \"a2bfb675-2581-4e1b-a305-215a157b86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:20.556667 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.556620 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" event={"ID":"13833b5b-ab9b-4a32-811b-5eb07645cfa9","Type":"ContainerStarted","Data":"f8e03bf279a58db19495cecf07019a8e99d7c23180af956907f2b935b9429f6f"} Oct 30 16:38:20.556846 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.556692 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" event={"ID":"13833b5b-ab9b-4a32-811b-5eb07645cfa9","Type":"ContainerStarted","Data":"b096f40c06100a9bed18f65c3231846bc8b2f56c85fe41e769d91365e6f8352f"} Oct 30 16:38:20.560354 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.560318 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" event={"ID":"1c251b61-eafd-43e1-b6b6-a91f4ba965ff","Type":"ContainerStarted","Data":"506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192"} Oct 30 16:38:20.560569 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.560550 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" event={"ID":"1c251b61-eafd-43e1-b6b6-a91f4ba965ff","Type":"ContainerStarted","Data":"0a8054e196490ff0da9246b5f5b697de182d7eea83ee7637182a3b54fcd74126"} Oct 30 16:38:20.580331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.580293 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aeddee8-bsg9h\" (UID: \"a2bfb675-2581-4e1b-a305-215a157b86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:20.580331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.580339 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zzcb\" (UniqueName: \"kubernetes.io/projected/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-kube-api-access-8zzcb\") pod \"maas-api-revoke-sub-1e604791-dxxwd\" (UID: \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:20.580536 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.580458 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aeddee8-bsg9h\" (UID: \"a2bfb675-2581-4e1b-a305-215a157b86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:20.580626 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.580591 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl2cn\" (UniqueName: \"kubernetes.io/projected/a2bfb675-2581-4e1b-a305-215a157b86dd-kube-api-access-tl2cn\") pod \"maas-api-revoke-sub-2aeddee8-bsg9h\" (UID: \"a2bfb675-2581-4e1b-a305-215a157b86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:20.580848 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.580827 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e604791-dxxwd\" (UID: \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:20.580906 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.580876 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e604791-dxxwd\" (UID: \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:20.581151 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.581128 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aeddee8-bsg9h\" (UID: \"a2bfb675-2581-4e1b-a305-215a157b86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:20.582946 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.582920 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aeddee8-bsg9h\" (UID: \"a2bfb675-2581-4e1b-a305-215a157b86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:20.602383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.601672 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" podStartSLOduration=2.601651838 podStartE2EDuration="2.601651838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:20.600125634 +0000 UTC m=+1062.348715022" watchObservedRunningTime="2026-10-30 16:38:20.601651838 +0000 UTC m=+1062.350241229" Oct 30 16:38:20.607590 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.607560 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl2cn\" (UniqueName: \"kubernetes.io/projected/a2bfb675-2581-4e1b-a305-215a157b86dd-kube-api-access-tl2cn\") pod \"maas-api-revoke-sub-2aeddee8-bsg9h\" (UID: \"a2bfb675-2581-4e1b-a305-215a157b86dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:20.657814 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.657681 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" podStartSLOduration=2.657659255 podStartE2EDuration="2.657659255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:20.646643917 +0000 UTC m=+1062.395233324" watchObservedRunningTime="2026-10-30 16:38:20.657659255 +0000 UTC m=+1062.406248643" Oct 30 16:38:20.683173 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.683123 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e604791-dxxwd\" (UID: \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:20.683349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.683183 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e604791-dxxwd\" (UID: \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:20.683349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.683343 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zzcb\" (UniqueName: \"kubernetes.io/projected/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-kube-api-access-8zzcb\") pod \"maas-api-revoke-sub-1e604791-dxxwd\" (UID: \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:20.684246 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.684220 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e604791-dxxwd\" (UID: \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:20.686521 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.686499 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e604791-dxxwd\" (UID: \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:20.694649 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.694628 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:20.707194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.707167 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zzcb\" (UniqueName: \"kubernetes.io/projected/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-kube-api-access-8zzcb\") pod \"maas-api-revoke-sub-1e604791-dxxwd\" (UID: \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:20.785774 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:20.785735 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:21.030002 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:21.027452 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:21.054994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:21.053960 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:38:21.094995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:21.058698 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" podUID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 16:38:21.473684 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:21.473628 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd"] Oct 30 16:38:21.475334 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:38:21.475304 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2418c9f_2c7e_4ed8_9c3e_443c88a6c6af.slice/crio-87535dbc43698271be5a4e3ee66711013682182fa400cce01357ec7d6cfce566 WatchSource:0}: Error finding container 87535dbc43698271be5a4e3ee66711013682182fa400cce01357ec7d6cfce566: Status 404 returned error can't find the container with id 87535dbc43698271be5a4e3ee66711013682182fa400cce01357ec7d6cfce566 Oct 30 16:38:21.477755 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:38:21.477722 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bfb675_2581_4e1b_a305_215a157b86dd.slice/crio-0a397885344cd297133be2e122627574d043e7ba362f01c4e7ec33b79d897cec WatchSource:0}: Error finding container 0a397885344cd297133be2e122627574d043e7ba362f01c4e7ec33b79d897cec: Status 404 returned error can't find the container with id 0a397885344cd297133be2e122627574d043e7ba362f01c4e7ec33b79d897cec Oct 30 16:38:21.482630 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:21.482601 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h"] Oct 30 16:38:21.493326 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:21.493250 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:21.494550 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:21.494518 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:38:21.494665 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:21.494574 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" podUID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 16:38:21.584047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:21.584006 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" event={"ID":"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af","Type":"ContainerStarted","Data":"87535dbc43698271be5a4e3ee66711013682182fa400cce01357ec7d6cfce566"} Oct 30 16:38:21.585547 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:21.585516 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" event={"ID":"a2bfb675-2581-4e1b-a305-215a157b86dd","Type":"ContainerStarted","Data":"0a397885344cd297133be2e122627574d043e7ba362f01c4e7ec33b79d897cec"} Oct 30 16:38:22.007822 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.007784 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:38:22.008265 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.007853 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" podUID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 16:38:22.377406 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.377315 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8"] Oct 30 16:38:22.383748 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.383719 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:22.386715 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.386690 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8"] Oct 30 16:38:22.487026 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.486960 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30626ac9-56ht8\" (UID: \"6aa987b7-701a-47b6-8c36-d59787846fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:22.487230 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.487204 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrb8z\" (UniqueName: \"kubernetes.io/projected/6aa987b7-701a-47b6-8c36-d59787846fbb-kube-api-access-hrb8z\") pod \"maas-api-revoke-sub-30626ac9-56ht8\" (UID: \"6aa987b7-701a-47b6-8c36-d59787846fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:22.487379 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.487356 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30626ac9-56ht8\" (UID: \"6aa987b7-701a-47b6-8c36-d59787846fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:22.493807 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.493635 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:38:22.493807 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.493681 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" podUID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 16:38:22.589037 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.588992 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30626ac9-56ht8\" (UID: \"6aa987b7-701a-47b6-8c36-d59787846fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:22.589214 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.589188 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrb8z\" (UniqueName: \"kubernetes.io/projected/6aa987b7-701a-47b6-8c36-d59787846fbb-kube-api-access-hrb8z\") pod \"maas-api-revoke-sub-30626ac9-56ht8\" (UID: \"6aa987b7-701a-47b6-8c36-d59787846fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:22.589304 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.589258 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30626ac9-56ht8\" (UID: \"6aa987b7-701a-47b6-8c36-d59787846fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:22.590054 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.590017 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30626ac9-56ht8\" (UID: \"6aa987b7-701a-47b6-8c36-d59787846fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:22.591475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.591451 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30626ac9-56ht8\" (UID: \"6aa987b7-701a-47b6-8c36-d59787846fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:22.611580 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.611538 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" event={"ID":"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af","Type":"ContainerStarted","Data":"14941ad30a3852f02b01a98ef8f6878151035a0e089b7bc27f7d781f6d861ecc"} Oct 30 16:38:22.660053 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.639540 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" event={"ID":"a2bfb675-2581-4e1b-a305-215a157b86dd","Type":"ContainerStarted","Data":"8a505c2b7eb38e091d5b59a61c2e5ab2b7f06e6b44c871623fb1a10404952416"} Oct 30 16:38:22.674492 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.674436 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrb8z\" (UniqueName: \"kubernetes.io/projected/6aa987b7-701a-47b6-8c36-d59787846fbb-kube-api-access-hrb8z\") pod \"maas-api-revoke-sub-30626ac9-56ht8\" (UID: \"6aa987b7-701a-47b6-8c36-d59787846fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:22.706567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.706536 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:22.709898 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.709848 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" podStartSLOduration=2.709835455 podStartE2EDuration="2.709835455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:22.706614753 +0000 UTC m=+1064.455204141" watchObservedRunningTime="2026-10-30 16:38:22.709835455 +0000 UTC m=+1064.458424842" Oct 30 16:38:22.710733 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:22.710695 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" podStartSLOduration=2.71068261 podStartE2EDuration="2.71068261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:22.667659271 +0000 UTC m=+1064.416248658" watchObservedRunningTime="2026-10-30 16:38:22.71068261 +0000 UTC m=+1064.459271997" Oct 30 16:38:23.031361 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:23.031278 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:38:23.031361 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:23.031337 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" podUID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 16:38:23.374125 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:38:23.374084 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa987b7_701a_47b6_8c36_d59787846fbb.slice/crio-7a08a5d5e3a81691bd2061d8028de84a36928138a53e946e6c66f152b54385e6 WatchSource:0}: Error finding container 7a08a5d5e3a81691bd2061d8028de84a36928138a53e946e6c66f152b54385e6: Status 404 returned error can't find the container with id 7a08a5d5e3a81691bd2061d8028de84a36928138a53e946e6c66f152b54385e6 Oct 30 16:38:23.376048 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:23.376020 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8"] Oct 30 16:38:23.493789 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:23.493746 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:38:23.494013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:23.493802 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" podUID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 16:38:23.589898 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:23.589839 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" podUID="74240496-5b00-44b2-a817-943efddfd6f9" containerName="payload-pre-processing" containerID="cri-o://eeac892295ea2b9f8345356d80aaf6ed8830710d7b60eed422af8d3ee5e18c61" gracePeriod=25 Oct 30 16:38:23.672829 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:23.672712 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" event={"ID":"6aa987b7-701a-47b6-8c36-d59787846fbb","Type":"ContainerStarted","Data":"7a08a5d5e3a81691bd2061d8028de84a36928138a53e946e6c66f152b54385e6"} Oct 30 16:38:23.675432 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:23.675397 5838 generic.go:356] "Generic (PLEG): container finished" podID="f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af" containerID="14941ad30a3852f02b01a98ef8f6878151035a0e089b7bc27f7d781f6d861ecc" exitCode=0 Oct 30 16:38:23.675645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:23.675587 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" event={"ID":"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af","Type":"ContainerDied","Data":"14941ad30a3852f02b01a98ef8f6878151035a0e089b7bc27f7d781f6d861ecc"} Oct 30 16:38:23.677438 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:23.677419 5838 generic.go:356] "Generic (PLEG): container finished" podID="a2bfb675-2581-4e1b-a305-215a157b86dd" containerID="8a505c2b7eb38e091d5b59a61c2e5ab2b7f06e6b44c871623fb1a10404952416" exitCode=0 Oct 30 16:38:23.677707 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:23.677582 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" event={"ID":"a2bfb675-2581-4e1b-a305-215a157b86dd","Type":"ContainerDied","Data":"8a505c2b7eb38e091d5b59a61c2e5ab2b7f06e6b44c871623fb1a10404952416"} Oct 30 16:38:24.008007 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:24.007879 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:38:24.008007 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:24.007937 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" podUID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 16:38:24.493638 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:24.493605 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:38:24.494056 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:24.493658 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" podUID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 16:38:24.786703 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:24.786662 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v"] Oct 30 16:38:24.835015 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:24.812197 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:24.950131 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:24.944585 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v\" (UID: \"d712e413-514d-4576-ae0a-c80e207ecab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:24.950131 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:24.944713 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v\" (UID: \"d712e413-514d-4576-ae0a-c80e207ecab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:24.950131 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:24.944892 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2wk5\" (UniqueName: \"kubernetes.io/projected/d712e413-514d-4576-ae0a-c80e207ecab7-kube-api-access-k2wk5\") pod \"maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v\" (UID: \"d712e413-514d-4576-ae0a-c80e207ecab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:24.991772 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:24.991741 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v"] Oct 30 16:38:24.996560 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:24.996521 5838 generic.go:356] "Generic (PLEG): container finished" podID="6aa987b7-701a-47b6-8c36-d59787846fbb" containerID="d5a8973969240e1e15a19fda55b42a518f687d748e210974cd5b9a2a57692fff" exitCode=0 Oct 30 16:38:24.999699 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:24.998471 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" event={"ID":"6aa987b7-701a-47b6-8c36-d59787846fbb","Type":"ContainerDied","Data":"d5a8973969240e1e15a19fda55b42a518f687d748e210974cd5b9a2a57692fff"} Oct 30 16:38:25.010126 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.010090 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:38:25.010414 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.010381 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" podUID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 16:38:25.061032 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.055897 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v\" (UID: \"d712e413-514d-4576-ae0a-c80e207ecab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:25.061032 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.056055 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v\" (UID: \"d712e413-514d-4576-ae0a-c80e207ecab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:25.061032 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.056246 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2wk5\" (UniqueName: \"kubernetes.io/projected/d712e413-514d-4576-ae0a-c80e207ecab7-kube-api-access-k2wk5\") pod \"maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v\" (UID: \"d712e413-514d-4576-ae0a-c80e207ecab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:25.061032 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.058790 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v\" (UID: \"d712e413-514d-4576-ae0a-c80e207ecab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:25.084253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.084215 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2wk5\" (UniqueName: \"kubernetes.io/projected/d712e413-514d-4576-ae0a-c80e207ecab7-kube-api-access-k2wk5\") pod \"maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v\" (UID: \"d712e413-514d-4576-ae0a-c80e207ecab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:25.087145 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.087105 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v\" (UID: \"d712e413-514d-4576-ae0a-c80e207ecab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:25.165297 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.165251 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:25.413217 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.412840 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:25.489166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.481965 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-xks-ca\") pod \"a2bfb675-2581-4e1b-a305-215a157b86dd\" (UID: \"a2bfb675-2581-4e1b-a305-215a157b86dd\") " Oct 30 16:38:25.489166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.482066 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2bfb675-2581-4e1b-a305-215a157b86dd-kube-api-access-tl2cn\" (UniqueName: \"kubernetes.io/projected/a2bfb675-2581-4e1b-a305-215a157b86dd-kube-api-access-tl2cn\") pod \"a2bfb675-2581-4e1b-a305-215a157b86dd\" (UID: \"a2bfb675-2581-4e1b-a305-215a157b86dd\") " Oct 30 16:38:25.489166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.482636 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-openshift-ca\") pod \"a2bfb675-2581-4e1b-a305-215a157b86dd\" (UID: \"a2bfb675-2581-4e1b-a305-215a157b86dd\") " Oct 30 16:38:25.489166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.482993 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-openshift-ca" pod "a2bfb675-2581-4e1b-a305-215a157b86dd" (UID: "a2bfb675-2581-4e1b-a305-215a157b86dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:25.489166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.485071 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-xks-ca" pod "a2bfb675-2581-4e1b-a305-215a157b86dd" (UID: "a2bfb675-2581-4e1b-a305-215a157b86dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:25.489166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.487476 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:25.489166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.487512 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bfb675-2581-4e1b-a305-215a157b86dd-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:25.489166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.488241 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2bfb675-2581-4e1b-a305-215a157b86dd-kube-api-access-tl2cn" pod "a2bfb675-2581-4e1b-a305-215a157b86dd" (UID: "a2bfb675-2581-4e1b-a305-215a157b86dd"). InnerVolumeSpecName "kube-api-access-tl2cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:25.490606 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.490578 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:25.494724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.494697 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:38:25.495142 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.494750 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" podUID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 16:38:25.585014 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.584930 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v"] Oct 30 16:38:25.586511 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:38:25.586457 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd712e413_514d_4576_ae0a_c80e207ecab7.slice/crio-5181cea243943a53ba458f879f28ade31466d1b680e56938aa8ef650d6f03bcd WatchSource:0}: Error finding container 5181cea243943a53ba458f879f28ade31466d1b680e56938aa8ef650d6f03bcd: Status 404 returned error can't find the container with id 5181cea243943a53ba458f879f28ade31466d1b680e56938aa8ef650d6f03bcd Oct 30 16:38:25.588215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.588193 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-kube-api-access-8zzcb\" (UniqueName: \"kubernetes.io/projected/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-kube-api-access-8zzcb\") pod \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\" (UID: \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\") " Oct 30 16:38:25.588419 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.588408 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-openshift-ca\") pod \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\" (UID: \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\") " Oct 30 16:38:25.588880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.588860 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-xks-ca\") pod \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\" (UID: \"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af\") " Oct 30 16:38:25.589406 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.589388 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl2cn\" (UniqueName: \"kubernetes.io/projected/a2bfb675-2581-4e1b-a305-215a157b86dd-kube-api-access-tl2cn\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:25.589905 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.589882 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-openshift-ca" pod "f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af" (UID: "f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:25.590745 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.590698 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-kube-api-access-8zzcb" pod "f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af" (UID: "f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af"). InnerVolumeSpecName "kube-api-access-8zzcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:25.591408 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.591387 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-xks-ca" pod "f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af" (UID: "f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:25.690752 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.690719 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:25.690752 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.690747 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zzcb\" (UniqueName: \"kubernetes.io/projected/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-kube-api-access-8zzcb\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:25.690752 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:25.690757 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:26.007604 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.007572 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 16:38:26.007778 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.007631 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" podUID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 16:38:26.008932 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.008912 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" Oct 30 16:38:26.009113 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.009088 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd" event={"ID":"f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af","Type":"ContainerDied","Data":"87535dbc43698271be5a4e3ee66711013682182fa400cce01357ec7d6cfce566"} Oct 30 16:38:26.009296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.009277 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87535dbc43698271be5a4e3ee66711013682182fa400cce01357ec7d6cfce566" Oct 30 16:38:26.010673 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.010657 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" Oct 30 16:38:26.010925 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.010694 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h" event={"ID":"a2bfb675-2581-4e1b-a305-215a157b86dd","Type":"ContainerDied","Data":"0a397885344cd297133be2e122627574d043e7ba362f01c4e7ec33b79d897cec"} Oct 30 16:38:26.010925 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.010895 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a397885344cd297133be2e122627574d043e7ba362f01c4e7ec33b79d897cec" Oct 30 16:38:26.012627 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.012567 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" event={"ID":"d712e413-514d-4576-ae0a-c80e207ecab7","Type":"ContainerStarted","Data":"5181cea243943a53ba458f879f28ade31466d1b680e56938aa8ef650d6f03bcd"} Oct 30 16:38:26.356019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.355374 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:26.403084 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.402205 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-xks-ca\") pod \"6aa987b7-701a-47b6-8c36-d59787846fbb\" (UID: \"6aa987b7-701a-47b6-8c36-d59787846fbb\") " Oct 30 16:38:26.403084 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.402469 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-openshift-ca\") pod \"6aa987b7-701a-47b6-8c36-d59787846fbb\" (UID: \"6aa987b7-701a-47b6-8c36-d59787846fbb\") " Oct 30 16:38:26.403084 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.402565 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa987b7-701a-47b6-8c36-d59787846fbb-kube-api-access-hrb8z\" (UniqueName: \"kubernetes.io/projected/6aa987b7-701a-47b6-8c36-d59787846fbb-kube-api-access-hrb8z\") pod \"6aa987b7-701a-47b6-8c36-d59787846fbb\" (UID: \"6aa987b7-701a-47b6-8c36-d59787846fbb\") " Oct 30 16:38:26.405104 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.403911 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-openshift-ca" pod "6aa987b7-701a-47b6-8c36-d59787846fbb" (UID: "6aa987b7-701a-47b6-8c36-d59787846fbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:26.417005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.414650 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa987b7-701a-47b6-8c36-d59787846fbb-kube-api-access-hrb8z" pod "6aa987b7-701a-47b6-8c36-d59787846fbb" (UID: "6aa987b7-701a-47b6-8c36-d59787846fbb"). InnerVolumeSpecName "kube-api-access-hrb8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:26.417005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.416820 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-xks-ca" pod "6aa987b7-701a-47b6-8c36-d59787846fbb" (UID: "6aa987b7-701a-47b6-8c36-d59787846fbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:26.493500 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.493466 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:38:26.493657 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.493524 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" podUID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 16:38:26.505425 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.505374 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:26.505896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.505429 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrb8z\" (UniqueName: \"kubernetes.io/projected/6aa987b7-701a-47b6-8c36-d59787846fbb-kube-api-access-hrb8z\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:26.505896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:26.505449 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa987b7-701a-47b6-8c36-d59787846fbb-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:27.027934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:27.027826 5838 generic.go:356] "Generic (PLEG): container finished" podID="d712e413-514d-4576-ae0a-c80e207ecab7" containerID="e198e48d3d15bea096ed60df6822e10ff87319f1550788ba12c255f0acf92ebc" exitCode=0 Oct 30 16:38:27.028129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:27.028099 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" event={"ID":"d712e413-514d-4576-ae0a-c80e207ecab7","Type":"ContainerDied","Data":"e198e48d3d15bea096ed60df6822e10ff87319f1550788ba12c255f0acf92ebc"} Oct 30 16:38:27.030034 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:27.029961 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" Oct 30 16:38:27.030276 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:27.030019 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8" event={"ID":"6aa987b7-701a-47b6-8c36-d59787846fbb","Type":"ContainerDied","Data":"7a08a5d5e3a81691bd2061d8028de84a36928138a53e946e6c66f152b54385e6"} Oct 30 16:38:27.030372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:27.030304 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a08a5d5e3a81691bd2061d8028de84a36928138a53e946e6c66f152b54385e6" Oct 30 16:38:27.457160 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:27.457068 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:27.457748 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:27.457708 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:27.462610 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:27.462576 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:38:27.494386 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:27.494341 5838 patch_prober.go:29] interesting pod/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:38:27.494626 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:27.494439 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" podUID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 16:38:28.217168 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.217137 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:28.242932 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.242905 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d712e413-514d-4576-ae0a-c80e207ecab7-kube-api-access-k2wk5\" (UniqueName: \"kubernetes.io/projected/d712e413-514d-4576-ae0a-c80e207ecab7-kube-api-access-k2wk5\") pod \"d712e413-514d-4576-ae0a-c80e207ecab7\" (UID: \"d712e413-514d-4576-ae0a-c80e207ecab7\") " Oct 30 16:38:28.243076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.243059 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-openshift-ca\") pod \"d712e413-514d-4576-ae0a-c80e207ecab7\" (UID: \"d712e413-514d-4576-ae0a-c80e207ecab7\") " Oct 30 16:38:28.243208 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.243191 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-xks-ca\") pod \"d712e413-514d-4576-ae0a-c80e207ecab7\" (UID: \"d712e413-514d-4576-ae0a-c80e207ecab7\") " Oct 30 16:38:28.243416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.243390 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-openshift-ca" pod "d712e413-514d-4576-ae0a-c80e207ecab7" (UID: "d712e413-514d-4576-ae0a-c80e207ecab7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:28.243882 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.243865 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:28.245089 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.245066 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d712e413-514d-4576-ae0a-c80e207ecab7-kube-api-access-k2wk5" pod "d712e413-514d-4576-ae0a-c80e207ecab7" (UID: "d712e413-514d-4576-ae0a-c80e207ecab7"). InnerVolumeSpecName "kube-api-access-k2wk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:28.245155 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.245138 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-xks-ca" pod "d712e413-514d-4576-ae0a-c80e207ecab7" (UID: "d712e413-514d-4576-ae0a-c80e207ecab7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:28.345728 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.345653 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2wk5\" (UniqueName: \"kubernetes.io/projected/d712e413-514d-4576-ae0a-c80e207ecab7-kube-api-access-k2wk5\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:28.345728 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.345682 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d712e413-514d-4576-ae0a-c80e207ecab7-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:28.500093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.500056 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:28.500903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.500882 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:28.501762 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.501736 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:38:28.816134 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.816075 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" containerID="cri-o://b3b8f18939d2d512575a8e524bb9e589a890863cef4f0586045119c83cbe65d9" gracePeriod=45 Oct 30 16:38:28.848135 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:28.848094 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" containerID="cri-o://1f6eb8f9e2cee3bcb5b07e6163f231d74993b4858d7a7fc5da733d3f26e6032e" gracePeriod=45 Oct 30 16:38:29.055123 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.055091 5838 generic.go:356] "Generic (PLEG): container finished" podID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerID="1f6eb8f9e2cee3bcb5b07e6163f231d74993b4858d7a7fc5da733d3f26e6032e" exitCode=0 Oct 30 16:38:29.055330 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.055176 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" event={"ID":"c2511107-80b5-4a19-9907-880a96e9b5f2","Type":"ContainerDied","Data":"1f6eb8f9e2cee3bcb5b07e6163f231d74993b4858d7a7fc5da733d3f26e6032e"} Oct 30 16:38:29.057107 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.057079 5838 generic.go:356] "Generic (PLEG): container finished" podID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerID="b3b8f18939d2d512575a8e524bb9e589a890863cef4f0586045119c83cbe65d9" exitCode=0 Oct 30 16:38:29.057252 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.057159 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" event={"ID":"860612c0-368d-4e00-9b59-7b4cdbfe0c2d","Type":"ContainerDied","Data":"b3b8f18939d2d512575a8e524bb9e589a890863cef4f0586045119c83cbe65d9"} Oct 30 16:38:29.058901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.058881 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" Oct 30 16:38:29.059096 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.058919 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v" event={"ID":"d712e413-514d-4576-ae0a-c80e207ecab7","Type":"ContainerDied","Data":"5181cea243943a53ba458f879f28ade31466d1b680e56938aa8ef650d6f03bcd"} Oct 30 16:38:29.059096 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.058964 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5181cea243943a53ba458f879f28ade31466d1b680e56938aa8ef650d6f03bcd" Oct 30 16:38:29.120872 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.120841 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:38:29.145014 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.144960 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:38:29.222746 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.222711 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tmp-dir\") pod \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " Oct 30 16:38:29.223275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.222776 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-kube-api-access-7g9kb\" (UniqueName: \"kubernetes.io/projected/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-kube-api-access-7g9kb\") pod \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " Oct 30 16:38:29.223275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.222798 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tls-certs\" (UniqueName: \"kubernetes.io/secret/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tls-certs\") pod \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " Oct 30 16:38:29.223275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.222838 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-home\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-home\") pod \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " Oct 30 16:38:29.223275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.222953 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-dshm\") pod \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " Oct 30 16:38:29.223275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.223023 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-model-cache\") pod \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\" (UID: \"860612c0-368d-4e00-9b59-7b4cdbfe0c2d\") " Oct 30 16:38:29.224314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.223676 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tmp-dir" pod "860612c0-368d-4e00-9b59-7b4cdbfe0c2d" (UID: "860612c0-368d-4e00-9b59-7b4cdbfe0c2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:29.224314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.223935 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-model-cache" pod "860612c0-368d-4e00-9b59-7b4cdbfe0c2d" (UID: "860612c0-368d-4e00-9b59-7b4cdbfe0c2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:29.224314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.224165 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-home" pod "860612c0-368d-4e00-9b59-7b4cdbfe0c2d" (UID: "860612c0-368d-4e00-9b59-7b4cdbfe0c2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:29.224770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.224719 5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-model-cache\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:29.224770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.224751 5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tmp-dir\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:29.224770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.224764 5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-home\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:29.225818 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.225726 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tls-certs" pod "860612c0-368d-4e00-9b59-7b4cdbfe0c2d" (UID: "860612c0-368d-4e00-9b59-7b4cdbfe0c2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:29.226335 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.226310 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-kube-api-access-7g9kb" pod "860612c0-368d-4e00-9b59-7b4cdbfe0c2d" (UID: "860612c0-368d-4e00-9b59-7b4cdbfe0c2d"). InnerVolumeSpecName "kube-api-access-7g9kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:29.226495 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.226481 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-dshm" pod "860612c0-368d-4e00-9b59-7b4cdbfe0c2d" (UID: "860612c0-368d-4e00-9b59-7b4cdbfe0c2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:29.330628 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.330576 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-dshm\") pod \"c2511107-80b5-4a19-9907-880a96e9b5f2\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " Oct 30 16:38:29.330837 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.330680 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-home\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-home\") pod \"c2511107-80b5-4a19-9907-880a96e9b5f2\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " Oct 30 16:38:29.330837 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.330802 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2511107-80b5-4a19-9907-880a96e9b5f2-kube-api-access-79pw4\" (UniqueName: \"kubernetes.io/projected/c2511107-80b5-4a19-9907-880a96e9b5f2-kube-api-access-79pw4\") pod \"c2511107-80b5-4a19-9907-880a96e9b5f2\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " Oct 30 16:38:29.330952 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.330931 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-tmp-dir\") pod \"c2511107-80b5-4a19-9907-880a96e9b5f2\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " Oct 30 16:38:29.331068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.331048 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-model-cache\") pod \"c2511107-80b5-4a19-9907-880a96e9b5f2\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " Oct 30 16:38:29.331187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.331139 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2511107-80b5-4a19-9907-880a96e9b5f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2511107-80b5-4a19-9907-880a96e9b5f2-tls-certs\") pod \"c2511107-80b5-4a19-9907-880a96e9b5f2\" (UID: \"c2511107-80b5-4a19-9907-880a96e9b5f2\") " Oct 30 16:38:29.332176 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.332150 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g9kb\" (UniqueName: \"kubernetes.io/projected/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-kube-api-access-7g9kb\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:29.332311 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.332179 5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-tls-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:29.332311 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.332194 5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860612c0-368d-4e00-9b59-7b4cdbfe0c2d-dshm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:29.333141 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.332882 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-tmp-dir" pod "c2511107-80b5-4a19-9907-880a96e9b5f2" (UID: "c2511107-80b5-4a19-9907-880a96e9b5f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:29.333141 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.332929 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-home" pod "c2511107-80b5-4a19-9907-880a96e9b5f2" (UID: "c2511107-80b5-4a19-9907-880a96e9b5f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:29.333414 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.333390 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-model-cache" pod "c2511107-80b5-4a19-9907-880a96e9b5f2" (UID: "c2511107-80b5-4a19-9907-880a96e9b5f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:29.334325 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.334252 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-dshm" pod "c2511107-80b5-4a19-9907-880a96e9b5f2" (UID: "c2511107-80b5-4a19-9907-880a96e9b5f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:29.334746 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.334723 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2511107-80b5-4a19-9907-880a96e9b5f2-tls-certs" pod "c2511107-80b5-4a19-9907-880a96e9b5f2" (UID: "c2511107-80b5-4a19-9907-880a96e9b5f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:29.334838 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.334815 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2511107-80b5-4a19-9907-880a96e9b5f2-kube-api-access-79pw4" pod "c2511107-80b5-4a19-9907-880a96e9b5f2" (UID: "c2511107-80b5-4a19-9907-880a96e9b5f2"). InnerVolumeSpecName "kube-api-access-79pw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:29.421114 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.421027 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw"] Oct 30 16:38:29.434197 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.434168 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79pw4\" (UniqueName: \"kubernetes.io/projected/c2511107-80b5-4a19-9907-880a96e9b5f2-kube-api-access-79pw4\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:29.434197 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.434195 5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-tmp-dir\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:29.434197 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.434207 5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-model-cache\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:29.434476 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.434214 5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2511107-80b5-4a19-9907-880a96e9b5f2-tls-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:29.434476 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.434222 5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-dshm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:29.434476 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.434229 5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2511107-80b5-4a19-9907-880a96e9b5f2-home\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:29.909194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.909153 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" podUID="02bc187b-d110-4d74-b2b9-2b96919c1a2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:38:29.909403 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.909268 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" podUID="2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:38:29.929043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:29.929005 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:38:30.069899 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.069862 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" event={"ID":"860612c0-368d-4e00-9b59-7b4cdbfe0c2d","Type":"ContainerDied","Data":"aa1ed311683d10757fb3b6f69c5cbf3305485bed05dca88478c585ef178649f9"} Oct 30 16:38:30.070092 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.069922 5838 scope.go:122] "RemoveContainer" containerID="b3b8f18939d2d512575a8e524bb9e589a890863cef4f0586045119c83cbe65d9" Oct 30 16:38:30.070244 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.070225 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg" Oct 30 16:38:30.073869 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.073840 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" Oct 30 16:38:30.074567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.074533 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" event={"ID":"c2511107-80b5-4a19-9907-880a96e9b5f2","Type":"ContainerDied","Data":"70f6b9b249c6928e5315c9465a550d87e641b6b7d45bb5dbc8f8394a03b4465c"} Oct 30 16:38:30.089322 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.089294 5838 scope.go:122] "RemoveContainer" containerID="1f6eb8f9e2cee3bcb5b07e6163f231d74993b4858d7a7fc5da733d3f26e6032e" Oct 30 16:38:30.137443 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.137404 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Oct 30 16:38:30.143022 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.142966 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn"] Oct 30 16:38:30.148419 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.148393 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3e4f91/distinct-2-w2-3e4f91-kserve-6d55c66988-vvrsn"] Oct 30 16:38:30.178923 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.178845 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg"] Oct 30 16:38:30.184290 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.184266 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3e4f91/premium-w2-3e4f91-kserve-58664cbc64-bcvpg"] Oct 30 16:38:30.428045 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.427957 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" path="/var/lib/kubelet/pods/860612c0-368d-4e00-9b59-7b4cdbfe0c2d/volumes" Oct 30 16:38:30.428750 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:30.428726 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" path="/var/lib/kubelet/pods/c2511107-80b5-4a19-9907-880a96e9b5f2/volumes" Oct 30 16:38:32.257263 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:32.097404 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:38:33.018997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.018102 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" podUID="74240496-5b00-44b2-a817-943efddfd6f9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:38:33.057012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.034123 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:33.321812 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.267365 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:33.373590 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.373540 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:33.521997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.518751 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:33.737058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.732849 5838 generic.go:356] "Generic (PLEG): container finished" podID="2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3" containerID="d4590db00d1a7ffe97be06b277da0c9db96b480e55b4a81b844e295353efc916" exitCode=0 Oct 30 16:38:33.737058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.733020 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" event={"ID":"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3","Type":"ContainerDied","Data":"d4590db00d1a7ffe97be06b277da0c9db96b480e55b4a81b844e295353efc916"} Oct 30 16:38:33.763039 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.763000 5838 generic.go:356] "Generic (PLEG): container finished" podID="02bc187b-d110-4d74-b2b9-2b96919c1a2a" containerID="b49659f70c245c3be13625c92279396aebe364884c4e4ee8905a335d9839237b" exitCode=0 Oct 30 16:38:33.771565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.771511 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" event={"ID":"02bc187b-d110-4d74-b2b9-2b96919c1a2a","Type":"ContainerDied","Data":"b49659f70c245c3be13625c92279396aebe364884c4e4ee8905a335d9839237b"} Oct 30 16:38:34.071576 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.923126 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:38:34.080440 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.995391 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:34.109231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.995428 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:34.141394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:33.995448 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:34.141394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.025647 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:34.141394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.052330 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 16:38:34.141394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.052424 5838 patch_prober.go:29] interesting pod/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:34.141394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.110793 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:34.141394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.066188 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" Oct 30 16:38:34.141394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.139004 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:38:34.249269 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.227640 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-plugins-config-volume\") pod \"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3\" (UID: \"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3\") " Oct 30 16:38:34.249269 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.227717 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-kube-api-access-fdczl\" (UniqueName: \"kubernetes.io/projected/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-kube-api-access-fdczl\") pod \"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3\" (UID: \"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3\") " Oct 30 16:38:34.252405 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.252036 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-plugins-config-volume" pod "2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3" (UID: "2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:34.281240 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.281208 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr"] Oct 30 16:38:34.283217 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.283179 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:34.298237 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.284363 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:38:34.630070 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.393874 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:34.630070 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.394074 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-kube-api-access-fdczl" pod "2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3" (UID: "2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3"). InnerVolumeSpecName "kube-api-access-fdczl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:34.630070 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.396370 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:38:34.630070 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.426388 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:34.767000 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:34.718426 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdczl\" (UniqueName: \"kubernetes.io/projected/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3-kube-api-access-fdczl\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:35.107014 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.099246 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 16:38:35.114998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.114430 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 16:38:35.140998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.125772 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:35.311427 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.311378 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:38:35.345051 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.342168 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" Oct 30 16:38:35.368093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.347781 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:35.403054 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.398321 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:35.542997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.541137 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:35.575334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.575267 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:35.801298 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.798349 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bc187b-d110-4d74-b2b9-2b96919c1a2a-kube-api-access-xpfxq\" (UniqueName: \"kubernetes.io/projected/02bc187b-d110-4d74-b2b9-2b96919c1a2a-kube-api-access-xpfxq\") pod \"02bc187b-d110-4d74-b2b9-2b96919c1a2a\" (UID: \"02bc187b-d110-4d74-b2b9-2b96919c1a2a\") " Oct 30 16:38:35.801298 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.798570 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02bc187b-d110-4d74-b2b9-2b96919c1a2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bc187b-d110-4d74-b2b9-2b96919c1a2a-plugins-config-volume\") pod \"02bc187b-d110-4d74-b2b9-2b96919c1a2a\" (UID: \"02bc187b-d110-4d74-b2b9-2b96919c1a2a\") " Oct 30 16:38:35.891274 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.874485 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bc187b-d110-4d74-b2b9-2b96919c1a2a-plugins-config-volume" pod "02bc187b-d110-4d74-b2b9-2b96919c1a2a" (UID: "02bc187b-d110-4d74-b2b9-2b96919c1a2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:35.893177 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.893046 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" Oct 30 16:38:35.948280 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.948221 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bc187b-d110-4d74-b2b9-2b96919c1a2a-kube-api-access-xpfxq" pod "02bc187b-d110-4d74-b2b9-2b96919c1a2a" (UID: "02bc187b-d110-4d74-b2b9-2b96919c1a2a"). InnerVolumeSpecName "kube-api-access-xpfxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:35.963004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.949856 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpfxq\" (UniqueName: \"kubernetes.io/projected/02bc187b-d110-4d74-b2b9-2b96919c1a2a-kube-api-access-xpfxq\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:35.963247 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:35.963221 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bc187b-d110-4d74-b2b9-2b96919c1a2a-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:36.029996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.004349 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" Oct 30 16:38:36.149104 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.125304 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:36.149104 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.125946 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:36.149104 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.132696 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:36.149104 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.147336 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 16:38:36.248028 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:36.233253 5838 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.525s" Oct 30 16:38:36.248028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.233377 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz" event={"ID":"2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3","Type":"ContainerDied","Data":"359d63ac743af59c597c60886d21e510a32617787263071655727086fba5ace7"} Oct 30 16:38:36.248028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.233445 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:38:36.248028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.233477 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-063dcc-l47tr"] Oct 30 16:38:36.248028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.233503 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj" event={"ID":"02bc187b-d110-4d74-b2b9-2b96919c1a2a","Type":"ContainerDied","Data":"d8c189faee617d31de9566055db10d3a1749dce504b62377e411f8f0919d00c4"} Oct 30 16:38:36.248028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.233559 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:38:36.248028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.233596 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:38:36.248028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.233631 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:38:36.248028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.233664 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:38:36.248028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.233698 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:38:36.248028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.233730 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:38:36.284711 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.284681 5838 scope.go:122] "RemoveContainer" containerID="d4590db00d1a7ffe97be06b277da0c9db96b480e55b4a81b844e295353efc916" Oct 30 16:38:36.314995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.314212 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:36.381013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.360114 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:36.401052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.390308 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:36.547544 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.547501 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:36.603060 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.602286 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:38:36.836255 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:36.833398 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e048cd-1dbc-4926-b951-2d5719660c41" path="/var/lib/kubelet/pods/f3e048cd-1dbc-4926-b951-2d5719660c41/volumes" Oct 30 16:38:37.146053 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.138771 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:37.146053 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.143701 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 16:38:37.146053 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.145192 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:38:37.171663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.154058 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p"] Oct 30 16:38:37.327199 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.327136 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:37.349123 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.349075 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz"] Oct 30 16:38:37.385130 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.368066 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:37.404584 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.404448 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": context deadline exceeded" Oct 30 16:38:37.422250 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.422196 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:38:37.709048 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.553210 5838 scope.go:122] "RemoveContainer" containerID="b49659f70c245c3be13625c92279396aebe364884c4e4ee8905a335d9839237b" Oct 30 16:38:37.714015 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.581085 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 16:38:37.718003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.649873 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:37.836498 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.836446 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m"] Oct 30 16:38:37.921946 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:37.921893 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.223996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:38.206071 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.223996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:38.206516 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.223996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:38.206826 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.223996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:38.209624 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.333351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:38.333242 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.414129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:38.388052 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:38.429716 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:38.429669 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": context deadline exceeded" Oct 30 16:38:38.723701 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:38.723558 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:38:38.734198 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:38.724533 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 16:38:39.288490 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.288442 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.317617 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.294637 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:38:39.560050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.531725 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.560050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.294675 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 16:38:39.560050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.534307 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:39.560050 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.558112 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.672078 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:39.657118 5838 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.258s" Oct 30 16:38:39.672078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.671102 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-063dcc-7cc7f8d89c-p4tjz"] Oct 30 16:38:39.672078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.671298 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:38:39.672078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.671342 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:38:39.672078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.671374 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:38:39.672078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.671406 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:38:39.672078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.671445 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:38:39.672078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.671476 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:38:39.672078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.671492 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz"] Oct 30 16:38:39.677065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.677040 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3e4f91-spqxz"] Oct 30 16:38:39.677289 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.677278 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:38:39.681452 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.681426 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:38:39.709105 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.709080 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj"] Oct 30 16:38:39.714838 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:39.714814 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-063dcc-6999c76fcf-l9qpj"] Oct 30 16:38:40.111444 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.111407 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755"] Oct 30 16:38:40.112131 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112114 5838 cpu_manager.go:414] "containerMap: removing container" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" Oct 30 16:38:40.112182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112131 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" Oct 30 16:38:40.112182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112156 5838 cpu_manager.go:414] "containerMap: removing container" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" Oct 30 16:38:40.112182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112161 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" Oct 30 16:38:40.112182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112171 5838 cpu_manager.go:414] "containerMap: removing container" podUID="02bc187b-d110-4d74-b2b9-2b96919c1a2a" containerName="payload-pre-processing" Oct 30 16:38:40.112182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112176 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bc187b-d110-4d74-b2b9-2b96919c1a2a" containerName="payload-pre-processing" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112185 5838 cpu_manager.go:414] "containerMap: removing container" podUID="2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3" containerName="payload-processing" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112190 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3" containerName="payload-processing" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112203 5838 cpu_manager.go:414] "containerMap: removing container" podUID="f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af" containerName="revoke-keys" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112209 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af" containerName="revoke-keys" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112216 5838 cpu_manager.go:414] "containerMap: removing container" podUID="a2bfb675-2581-4e1b-a305-215a157b86dd" containerName="revoke-keys" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112221 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2bfb675-2581-4e1b-a305-215a157b86dd" containerName="revoke-keys" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112227 5838 cpu_manager.go:414] "containerMap: removing container" podUID="6aa987b7-701a-47b6-8c36-d59787846fbb" containerName="revoke-keys" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112231 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa987b7-701a-47b6-8c36-d59787846fbb" containerName="revoke-keys" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112246 5838 cpu_manager.go:414] "containerMap: removing container" podUID="d712e413-514d-4576-ae0a-c80e207ecab7" containerName="revoke-keys" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112251 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d712e413-514d-4576-ae0a-c80e207ecab7" containerName="revoke-keys" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112345 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bc187b-d110-4d74-b2b9-2b96919c1a2a" containerName="payload-pre-processing" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112353 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3" containerName="payload-processing" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112364 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af" containerName="revoke-keys" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112371 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2bfb675-2581-4e1b-a305-215a157b86dd" containerName="revoke-keys" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112377 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa987b7-701a-47b6-8c36-d59787846fbb" containerName="revoke-keys" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112387 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d712e413-514d-4576-ae0a-c80e207ecab7" containerName="revoke-keys" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112396 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="860612c0-368d-4e00-9b59-7b4cdbfe0c2d" containerName="main" Oct 30 16:38:40.112697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.112405 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2511107-80b5-4a19-9907-880a96e9b5f2" containerName="main" Oct 30 16:38:40.118237 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.118215 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:40.132631 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.132599 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:40.132783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.132753 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:38:40.135788 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.135759 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755"] Oct 30 16:38:40.153506 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.153461 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b89a3a85-vr755\" (UID: \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:40.153702 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.153660 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fcnx\" (UniqueName: \"kubernetes.io/projected/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-kube-api-access-6fcnx\") pod \"maas-api-revoke-sub-b89a3a85-vr755\" (UID: \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:40.153776 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.153764 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b89a3a85-vr755\" (UID: \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:40.255885 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.255845 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b89a3a85-vr755\" (UID: \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:40.256096 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.255988 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fcnx\" (UniqueName: \"kubernetes.io/projected/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-kube-api-access-6fcnx\") pod \"maas-api-revoke-sub-b89a3a85-vr755\" (UID: \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:40.256096 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.256037 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b89a3a85-vr755\" (UID: \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:40.256610 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.256588 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b89a3a85-vr755\" (UID: \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:40.258733 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.258711 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b89a3a85-vr755\" (UID: \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:40.279212 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.279179 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fcnx\" (UniqueName: \"kubernetes.io/projected/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-kube-api-access-6fcnx\") pod \"maas-api-revoke-sub-b89a3a85-vr755\" (UID: \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:40.411688 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.409664 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02bc187b-d110-4d74-b2b9-2b96919c1a2a" path="/var/lib/kubelet/pods/02bc187b-d110-4d74-b2b9-2b96919c1a2a/volumes" Oct 30 16:38:40.411688 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.410373 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3" path="/var/lib/kubelet/pods/2a4edb3a-808c-4be8-8d9c-c96fb42f5ce3/volumes" Oct 30 16:38:40.411688 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.411037 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7acae368-f382-4336-b418-221bd12ebe8e" path="/var/lib/kubelet/pods/7acae368-f382-4336-b418-221bd12ebe8e/volumes" Oct 30 16:38:40.438121 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.434139 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:40.782550 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:40.782516 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755"] Oct 30 16:38:40.785745 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:38:40.785713 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0113a6c2_db8d_469c_aefa_0d8b5a9e321a.slice/crio-1af9ffd16c81341f281374cd18fc5fb682aaeead18a9a56daef2569b2ea41f5a WatchSource:0}: Error finding container 1af9ffd16c81341f281374cd18fc5fb682aaeead18a9a56daef2569b2ea41f5a: Status 404 returned error can't find the container with id 1af9ffd16c81341f281374cd18fc5fb682aaeead18a9a56daef2569b2ea41f5a Oct 30 16:38:41.710061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:41.710015 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" event={"ID":"0113a6c2-db8d-469c-aefa-0d8b5a9e321a","Type":"ContainerStarted","Data":"a91344d841478ea418fa68ea59da417c9d7ca55dd23b0bc8f3e9618eee81c2dd"} Oct 30 16:38:41.710546 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:41.710130 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" event={"ID":"0113a6c2-db8d-469c-aefa-0d8b5a9e321a","Type":"ContainerStarted","Data":"1af9ffd16c81341f281374cd18fc5fb682aaeead18a9a56daef2569b2ea41f5a"} Oct 30 16:38:41.809003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:41.789406 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" podStartSLOduration=1.7893816139999998 podStartE2EDuration="1.789381614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:41.756916869 +0000 UTC m=+1083.505506280" watchObservedRunningTime="2026-10-30 16:38:41.789381614 +0000 UTC m=+1083.537971026" Oct 30 16:38:41.841555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:41.841222 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn"] Oct 30 16:38:41.847464 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:41.847426 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:41.849108 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:41.849081 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn"] Oct 30 16:38:41.901926 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:41.901898 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec92c54e-cmknn\" (UID: \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:41.902228 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:41.902205 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec92c54e-cmknn\" (UID: \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:41.902363 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:41.902349 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knc9r\" (UniqueName: \"kubernetes.io/projected/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-kube-api-access-knc9r\") pod \"maas-api-revoke-sub-ec92c54e-cmknn\" (UID: \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:42.007774 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.007731 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knc9r\" (UniqueName: \"kubernetes.io/projected/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-kube-api-access-knc9r\") pod \"maas-api-revoke-sub-ec92c54e-cmknn\" (UID: \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:42.007963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.007912 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec92c54e-cmknn\" (UID: \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:42.008130 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.008112 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec92c54e-cmknn\" (UID: \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:42.008959 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.008931 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec92c54e-cmknn\" (UID: \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:42.015686 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.015652 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec92c54e-cmknn\" (UID: \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:42.029929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.029880 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4"] Oct 30 16:38:42.030182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.030148 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" containerID="cri-o://f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a" gracePeriod=30 Oct 30 16:38:42.037223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.037185 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg"] Oct 30 16:38:42.037486 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.037448 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" containerID="cri-o://e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced" gracePeriod=30 Oct 30 16:38:42.047481 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.047413 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p"] Oct 30 16:38:42.047687 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.047657 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" podUID="64654948-d621-4884-bb13-0501973a1d4f" containerName="istio-proxy" containerID="cri-o://2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a" gracePeriod=30 Oct 30 16:38:42.049030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.048961 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knc9r\" (UniqueName: \"kubernetes.io/projected/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-kube-api-access-knc9r\") pod \"maas-api-revoke-sub-ec92c54e-cmknn\" (UID: \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:42.054178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.054148 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn"] Oct 30 16:38:42.054535 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.054494 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" podUID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" containerName="istio-proxy" containerID="cri-o://8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c" gracePeriod=30 Oct 30 16:38:42.155863 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.155822 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" podUID="a1a40be2-e7d1-49f9-a304-2623278732ba" containerName="payload-processing" containerID="cri-o://73cccef9c8dece852b7aa5213fd03e58abe3387099d5f53e4cf4867aab0e7d08" gracePeriod=25 Oct 30 16:38:42.208673 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.208635 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:42.540049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.537201 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:38:42.540049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.537286 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" podUID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:38:42.630065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.621200 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:38:42.630065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.621267 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" podUID="64654948-d621-4884-bb13-0501973a1d4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:38:42.649002 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.645274 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn"] Oct 30 16:38:42.684642 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:38:42.684261 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc01ad1_1d4f_4ac5_8b7b_d7c582e13243.slice/crio-fedcccebc3f5f5f27de5004d4ea46fdd90286b64452cd25a43dec8eca7c97643 WatchSource:0}: Error finding container fedcccebc3f5f5f27de5004d4ea46fdd90286b64452cd25a43dec8eca7c97643: Status 404 returned error can't find the container with id fedcccebc3f5f5f27de5004d4ea46fdd90286b64452cd25a43dec8eca7c97643 Oct 30 16:38:42.722026 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.721942 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" event={"ID":"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243","Type":"ContainerStarted","Data":"fedcccebc3f5f5f27de5004d4ea46fdd90286b64452cd25a43dec8eca7c97643"} Oct 30 16:38:42.737100 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.737051 5838 generic.go:356] "Generic (PLEG): container finished" podID="0113a6c2-db8d-469c-aefa-0d8b5a9e321a" containerID="a91344d841478ea418fa68ea59da417c9d7ca55dd23b0bc8f3e9618eee81c2dd" exitCode=0 Oct 30 16:38:42.737269 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.737127 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" event={"ID":"0113a6c2-db8d-469c-aefa-0d8b5a9e321a","Type":"ContainerDied","Data":"a91344d841478ea418fa68ea59da417c9d7ca55dd23b0bc8f3e9618eee81c2dd"} Oct 30 16:38:42.837704 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.837603 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" podUID="0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98" containerName="payload-pre-processing" containerID="cri-o://e77f525c1869f3c4318c540dec0abfc3c9492e9b0506692786cdf96c5caeed5e" gracePeriod=25 Oct 30 16:38:42.952383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:42.952347 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" podUID="74240496-5b00-44b2-a817-943efddfd6f9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:38:43.749784 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.749749 5838 generic.go:356] "Generic (PLEG): container finished" podID="74240496-5b00-44b2-a817-943efddfd6f9" containerID="eeac892295ea2b9f8345356d80aaf6ed8830710d7b60eed422af8d3ee5e18c61" exitCode=0 Oct 30 16:38:43.750264 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.749830 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" event={"ID":"74240496-5b00-44b2-a817-943efddfd6f9","Type":"ContainerDied","Data":"eeac892295ea2b9f8345356d80aaf6ed8830710d7b60eed422af8d3ee5e18c61"} Oct 30 16:38:43.752244 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.752212 5838 generic.go:356] "Generic (PLEG): container finished" podID="9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243" containerID="e72d611265c2dd9bc2682ab9c2352cb3849c6b284dc5a79dbe671f44b115c484" exitCode=0 Oct 30 16:38:43.752399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.752327 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" event={"ID":"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243","Type":"ContainerDied","Data":"e72d611265c2dd9bc2682ab9c2352cb3849c6b284dc5a79dbe671f44b115c484"} Oct 30 16:38:43.907185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.907155 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" Oct 30 16:38:43.923392 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.923360 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74240496-5b00-44b2-a817-943efddfd6f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74240496-5b00-44b2-a817-943efddfd6f9-plugins-config-volume\") pod \"74240496-5b00-44b2-a817-943efddfd6f9\" (UID: \"74240496-5b00-44b2-a817-943efddfd6f9\") " Oct 30 16:38:43.923627 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.923555 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74240496-5b00-44b2-a817-943efddfd6f9-kube-api-access-lsvxv\" (UniqueName: \"kubernetes.io/projected/74240496-5b00-44b2-a817-943efddfd6f9-kube-api-access-lsvxv\") pod \"74240496-5b00-44b2-a817-943efddfd6f9\" (UID: \"74240496-5b00-44b2-a817-943efddfd6f9\") " Oct 30 16:38:43.925099 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.925067 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74240496-5b00-44b2-a817-943efddfd6f9-plugins-config-volume" pod "74240496-5b00-44b2-a817-943efddfd6f9" (UID: "74240496-5b00-44b2-a817-943efddfd6f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:43.928596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.928566 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74240496-5b00-44b2-a817-943efddfd6f9-kube-api-access-lsvxv" pod "74240496-5b00-44b2-a817-943efddfd6f9" (UID: "74240496-5b00-44b2-a817-943efddfd6f9"). InnerVolumeSpecName "kube-api-access-lsvxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:43.937038 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.936965 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd"] Oct 30 16:38:43.938134 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.938116 5838 cpu_manager.go:414] "containerMap: removing container" podUID="74240496-5b00-44b2-a817-943efddfd6f9" containerName="payload-pre-processing" Oct 30 16:38:43.938193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.938136 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74240496-5b00-44b2-a817-943efddfd6f9" containerName="payload-pre-processing" Oct 30 16:38:43.938323 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.938311 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="74240496-5b00-44b2-a817-943efddfd6f9" containerName="payload-pre-processing" Oct 30 16:38:43.943256 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.943233 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:43.975489 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.975455 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd"] Oct 30 16:38:43.988339 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:43.988307 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:44.025533 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.025497 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-kube-api-access-6fcnx\" (UniqueName: \"kubernetes.io/projected/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-kube-api-access-6fcnx\") pod \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\" (UID: \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\") " Oct 30 16:38:44.025726 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.025680 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-xks-ca\") pod \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\" (UID: \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\") " Oct 30 16:38:44.025788 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.025731 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-openshift-ca\") pod \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\" (UID: \"0113a6c2-db8d-469c-aefa-0d8b5a9e321a\") " Oct 30 16:38:44.026218 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.026185 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-openshift-ca" pod "0113a6c2-db8d-469c-aefa-0d8b5a9e321a" (UID: "0113a6c2-db8d-469c-aefa-0d8b5a9e321a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:44.026359 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.026266 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc5d5d7-cf9cd\" (UID: \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:44.026494 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.026438 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc5d5d7-cf9cd\" (UID: \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:44.026769 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.026744 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9j7k\" (UniqueName: \"kubernetes.io/projected/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-kube-api-access-m9j7k\") pod \"maas-api-revoke-sub-6dc5d5d7-cf9cd\" (UID: \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:44.026956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.026939 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:44.027051 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.026962 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74240496-5b00-44b2-a817-943efddfd6f9-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:44.027051 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.026993 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsvxv\" (UniqueName: \"kubernetes.io/projected/74240496-5b00-44b2-a817-943efddfd6f9-kube-api-access-lsvxv\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:44.028374 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.028344 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-kube-api-access-6fcnx" pod "0113a6c2-db8d-469c-aefa-0d8b5a9e321a" (UID: "0113a6c2-db8d-469c-aefa-0d8b5a9e321a"). InnerVolumeSpecName "kube-api-access-6fcnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:44.032135 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.032095 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-xks-ca" pod "0113a6c2-db8d-469c-aefa-0d8b5a9e321a" (UID: "0113a6c2-db8d-469c-aefa-0d8b5a9e321a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:44.167254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.156908 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc5d5d7-cf9cd\" (UID: \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:44.167254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.157732 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc5d5d7-cf9cd\" (UID: \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:44.167254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.158083 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc5d5d7-cf9cd\" (UID: \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:44.167254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.158789 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9j7k\" (UniqueName: \"kubernetes.io/projected/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-kube-api-access-m9j7k\") pod \"maas-api-revoke-sub-6dc5d5d7-cf9cd\" (UID: \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:44.167254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.159234 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fcnx\" (UniqueName: \"kubernetes.io/projected/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-kube-api-access-6fcnx\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:44.167254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.159253 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0113a6c2-db8d-469c-aefa-0d8b5a9e321a-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:44.219994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.198358 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc5d5d7-cf9cd\" (UID: \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:44.235867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.235819 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9j7k\" (UniqueName: \"kubernetes.io/projected/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-kube-api-access-m9j7k\") pod \"maas-api-revoke-sub-6dc5d5d7-cf9cd\" (UID: \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:44.320276 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.302315 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:44.447229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.447176 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" containerID="cri-o://6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9" gracePeriod=45 Oct 30 16:38:44.682552 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.682460 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:38:44.682552 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.682518 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:38:44.690807 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.689136 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd"] Oct 30 16:38:44.763916 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.763880 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" Oct 30 16:38:44.764331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.763932 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755" event={"ID":"0113a6c2-db8d-469c-aefa-0d8b5a9e321a","Type":"ContainerDied","Data":"1af9ffd16c81341f281374cd18fc5fb682aaeead18a9a56daef2569b2ea41f5a"} Oct 30 16:38:44.764331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.764003 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1af9ffd16c81341f281374cd18fc5fb682aaeead18a9a56daef2569b2ea41f5a" Oct 30 16:38:44.765906 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.765875 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" event={"ID":"74240496-5b00-44b2-a817-943efddfd6f9","Type":"ContainerDied","Data":"6eb225190a64a32849ba2fa8c32a3210e5f85777f802cad822aa2a0509ce8b9e"} Oct 30 16:38:44.766102 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.765937 5838 scope.go:122] "RemoveContainer" containerID="eeac892295ea2b9f8345356d80aaf6ed8830710d7b60eed422af8d3ee5e18c61" Oct 30 16:38:44.766169 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.766155 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv" Oct 30 16:38:44.770193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.769493 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:38:44.770193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.770157 5838 generic.go:356] "Generic (PLEG): container finished" podID="da70d01c-6ab3-495f-8056-0546653f2a53" containerID="6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9" exitCode=0 Oct 30 16:38:44.770334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.770318 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" event={"ID":"da70d01c-6ab3-495f-8056-0546653f2a53","Type":"ContainerDied","Data":"6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9"} Oct 30 16:38:44.770376 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.770350 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" event={"ID":"da70d01c-6ab3-495f-8056-0546653f2a53","Type":"ContainerDied","Data":"24f1d5505fa0301d746c51ee8afbc85c748f1fe8a33af578c8a6d642fbdf98d1"} Oct 30 16:38:44.772236 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.772211 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" event={"ID":"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9","Type":"ContainerStarted","Data":"77699c53e74ddde06ff41d7f84705390e1e31ec2702889f4012d03a6c9cc0352"} Oct 30 16:38:44.783367 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.783341 5838 scope.go:122] "RemoveContainer" containerID="6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9" Oct 30 16:38:44.800314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.800288 5838 scope.go:122] "RemoveContainer" containerID="6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9" Oct 30 16:38:44.800705 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:44.800676 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9\": container with ID starting with 6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9 not found: ID does not exist" containerID="6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9" Oct 30 16:38:44.800822 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.800716 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9"} err="failed to get container status \"6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9\": rpc error: code = NotFound desc = could not find container \"6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9\": container with ID starting with 6ec72dc65c1674e251460e56dcc6351420ec42bebef25cb73b7a329099fcb5a9 not found: ID does not exist" Oct 30 16:38:44.862267 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.862231 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv"] Oct 30 16:38:44.866996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.866949 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3e4f91-bcd85c65f-n8xfv"] Oct 30 16:38:44.889383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.889331 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-model-cache\") pod \"da70d01c-6ab3-495f-8056-0546653f2a53\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " Oct 30 16:38:44.889585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.889400 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-tmp-dir\") pod \"da70d01c-6ab3-495f-8056-0546653f2a53\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " Oct 30 16:38:44.889661 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.889617 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da70d01c-6ab3-495f-8056-0546653f2a53-kube-api-access-kzhhm\" (UniqueName: \"kubernetes.io/projected/da70d01c-6ab3-495f-8056-0546653f2a53-kube-api-access-kzhhm\") pod \"da70d01c-6ab3-495f-8056-0546653f2a53\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " Oct 30 16:38:44.889727 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.889671 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-model-cache" pod "da70d01c-6ab3-495f-8056-0546653f2a53" (UID: "da70d01c-6ab3-495f-8056-0546653f2a53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:44.889727 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.889682 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-tmp-dir" pod "da70d01c-6ab3-495f-8056-0546653f2a53" (UID: "da70d01c-6ab3-495f-8056-0546653f2a53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:44.889840 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.889741 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-home\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-home\") pod \"da70d01c-6ab3-495f-8056-0546653f2a53\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " Oct 30 16:38:44.889840 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.889782 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da70d01c-6ab3-495f-8056-0546653f2a53-tls-certs\" (UniqueName: \"kubernetes.io/secret/da70d01c-6ab3-495f-8056-0546653f2a53-tls-certs\") pod \"da70d01c-6ab3-495f-8056-0546653f2a53\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " Oct 30 16:38:44.889940 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.889876 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-dshm\") pod \"da70d01c-6ab3-495f-8056-0546653f2a53\" (UID: \"da70d01c-6ab3-495f-8056-0546653f2a53\") " Oct 30 16:38:44.890267 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.890231 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-home" pod "da70d01c-6ab3-495f-8056-0546653f2a53" (UID: "da70d01c-6ab3-495f-8056-0546653f2a53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:44.890621 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.890557 5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-tmp-dir\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:44.890808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.890791 5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-model-cache\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:44.915353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.915307 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-dshm" pod "da70d01c-6ab3-495f-8056-0546653f2a53" (UID: "da70d01c-6ab3-495f-8056-0546653f2a53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:44.915885 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.915848 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da70d01c-6ab3-495f-8056-0546653f2a53-kube-api-access-kzhhm" pod "da70d01c-6ab3-495f-8056-0546653f2a53" (UID: "da70d01c-6ab3-495f-8056-0546653f2a53"). InnerVolumeSpecName "kube-api-access-kzhhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:44.917223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.917199 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da70d01c-6ab3-495f-8056-0546653f2a53-tls-certs" pod "da70d01c-6ab3-495f-8056-0546653f2a53" (UID: "da70d01c-6ab3-495f-8056-0546653f2a53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:44.923725 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.923685 5838 patch_prober.go:29] interesting pod/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:38:44.923857 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.923791 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:38:44.989961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.989933 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:44.992351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.992315 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzhhm\" (UniqueName: \"kubernetes.io/projected/da70d01c-6ab3-495f-8056-0546653f2a53-kube-api-access-kzhhm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:44.992351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.992351 5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-home\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:44.992537 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.992363 5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da70d01c-6ab3-495f-8056-0546653f2a53-tls-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:44.992537 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:44.992372 5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da70d01c-6ab3-495f-8056-0546653f2a53-dshm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:45.093496 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.093455 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-openshift-ca\") pod \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\" (UID: \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\") " Oct 30 16:38:45.093707 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.093656 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-xks-ca\") pod \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\" (UID: \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\") " Oct 30 16:38:45.093794 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.093757 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-kube-api-access-knc9r\" (UniqueName: \"kubernetes.io/projected/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-kube-api-access-knc9r\") pod \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\" (UID: \"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243\") " Oct 30 16:38:45.093899 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.093870 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-openshift-ca" pod "9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243" (UID: "9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:45.094557 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.094529 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:45.095873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.095847 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-xks-ca" pod "9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243" (UID: "9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:45.096403 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.096380 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-kube-api-access-knc9r" pod "9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243" (UID: "9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243"). InnerVolumeSpecName "kube-api-access-knc9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:45.196467 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.196428 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:45.196467 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.196461 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knc9r\" (UniqueName: \"kubernetes.io/projected/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243-kube-api-access-knc9r\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:45.782600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.782567 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" Oct 30 16:38:45.783107 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.782592 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn" event={"ID":"9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243","Type":"ContainerDied","Data":"fedcccebc3f5f5f27de5004d4ea46fdd90286b64452cd25a43dec8eca7c97643"} Oct 30 16:38:45.783107 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.782653 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fedcccebc3f5f5f27de5004d4ea46fdd90286b64452cd25a43dec8eca7c97643" Oct 30 16:38:45.785614 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.785587 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw" Oct 30 16:38:45.791353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.791310 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" event={"ID":"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9","Type":"ContainerStarted","Data":"823901dcf75c5dfc08eb189a542927860cf2abd2597ed4d17a87d9beadc9fa38"} Oct 30 16:38:45.854027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.853988 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw"] Oct 30 16:38:45.936047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.932751 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2e0784/distinct-2-w5-2e0784-kserve-755cbcf967-c99fw"] Oct 30 16:38:45.964608 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:45.964519 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" podStartSLOduration=2.964499156 podStartE2EDuration="2.964499156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:45.928414696 +0000 UTC m=+1087.677004084" watchObservedRunningTime="2026-10-30 16:38:45.964499156 +0000 UTC m=+1087.713088544" Oct 30 16:38:46.050109 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.050068 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr"] Oct 30 16:38:46.051966 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.050317 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" podUID="d212d906-4b33-44b1-9079-1b5ae0000f3b" containerName="istio-proxy" containerID="cri-o://aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96" gracePeriod=30 Oct 30 16:38:46.058474 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.058442 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw"] Oct 30 16:38:46.058711 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.058685 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" containerID="cri-o://84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213" gracePeriod=30 Oct 30 16:38:46.071595 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.071557 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch"] Oct 30 16:38:46.071851 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.071822 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerName="istio-proxy" containerID="cri-o://810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191" gracePeriod=30 Oct 30 16:38:46.127499 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.127467 5838 patch_prober.go:29] interesting pod/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:38:46.127659 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.127516 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:38:46.426418 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.426360 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74240496-5b00-44b2-a817-943efddfd6f9" path="/var/lib/kubelet/pods/74240496-5b00-44b2-a817-943efddfd6f9/volumes" Oct 30 16:38:46.427623 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.427592 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" path="/var/lib/kubelet/pods/da70d01c-6ab3-495f-8056-0546653f2a53/volumes" Oct 30 16:38:46.658216 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.657181 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg"] Oct 30 16:38:46.659749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.658772 5838 cpu_manager.go:414] "containerMap: removing container" podUID="0113a6c2-db8d-469c-aefa-0d8b5a9e321a" containerName="revoke-keys" Oct 30 16:38:46.659749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.658801 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0113a6c2-db8d-469c-aefa-0d8b5a9e321a" containerName="revoke-keys" Oct 30 16:38:46.659749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.658922 5838 cpu_manager.go:414] "containerMap: removing container" podUID="9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243" containerName="revoke-keys" Oct 30 16:38:46.659749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.658931 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243" containerName="revoke-keys" Oct 30 16:38:46.659749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.658951 5838 cpu_manager.go:414] "containerMap: removing container" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" Oct 30 16:38:46.659749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.658961 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" Oct 30 16:38:46.659749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.659300 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="0113a6c2-db8d-469c-aefa-0d8b5a9e321a" containerName="revoke-keys" Oct 30 16:38:46.659749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.659337 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243" containerName="revoke-keys" Oct 30 16:38:46.659749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.659352 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="da70d01c-6ab3-495f-8056-0546653f2a53" containerName="main" Oct 30 16:38:46.668067 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.668024 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:46.679725 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.679680 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg"] Oct 30 16:38:46.727413 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.727370 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee7dd1a4-68pzg\" (UID: \"555c9e26-cfa2-4413-81a5-f21a77718534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:46.727736 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.727682 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw84j\" (UniqueName: \"kubernetes.io/projected/555c9e26-cfa2-4413-81a5-f21a77718534-kube-api-access-vw84j\") pod \"maas-api-revoke-sub-ee7dd1a4-68pzg\" (UID: \"555c9e26-cfa2-4413-81a5-f21a77718534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:46.727840 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.727744 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee7dd1a4-68pzg\" (UID: \"555c9e26-cfa2-4413-81a5-f21a77718534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:46.801645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.801613 5838 generic.go:356] "Generic (PLEG): container finished" podID="f9bc4a05-8c71-4073-a2c3-779b80a3a3d9" containerID="823901dcf75c5dfc08eb189a542927860cf2abd2597ed4d17a87d9beadc9fa38" exitCode=0 Oct 30 16:38:46.802078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.801697 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" event={"ID":"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9","Type":"ContainerDied","Data":"823901dcf75c5dfc08eb189a542927860cf2abd2597ed4d17a87d9beadc9fa38"} Oct 30 16:38:46.829011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.828953 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw84j\" (UniqueName: \"kubernetes.io/projected/555c9e26-cfa2-4413-81a5-f21a77718534-kube-api-access-vw84j\") pod \"maas-api-revoke-sub-ee7dd1a4-68pzg\" (UID: \"555c9e26-cfa2-4413-81a5-f21a77718534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:46.829188 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.829025 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee7dd1a4-68pzg\" (UID: \"555c9e26-cfa2-4413-81a5-f21a77718534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:46.829188 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.829107 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee7dd1a4-68pzg\" (UID: \"555c9e26-cfa2-4413-81a5-f21a77718534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:46.829841 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.829820 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee7dd1a4-68pzg\" (UID: \"555c9e26-cfa2-4413-81a5-f21a77718534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:46.831461 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.831444 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee7dd1a4-68pzg\" (UID: \"555c9e26-cfa2-4413-81a5-f21a77718534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:46.851754 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.851723 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw84j\" (UniqueName: \"kubernetes.io/projected/555c9e26-cfa2-4413-81a5-f21a77718534-kube-api-access-vw84j\") pod \"maas-api-revoke-sub-ee7dd1a4-68pzg\" (UID: \"555c9e26-cfa2-4413-81a5-f21a77718534\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:46.986607 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:46.986571 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:47.303954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.303917 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg"] Oct 30 16:38:47.304393 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:38:47.304361 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555c9e26_cfa2_4413_81a5_f21a77718534.slice/crio-040fefd029105586caa1cfe99e8bf6bfe5311c008dc0923de5fce00aa0a47fab WatchSource:0}: Error finding container 040fefd029105586caa1cfe99e8bf6bfe5311c008dc0923de5fce00aa0a47fab: Status 404 returned error can't find the container with id 040fefd029105586caa1cfe99e8bf6bfe5311c008dc0923de5fce00aa0a47fab Oct 30 16:38:47.364781 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.364744 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:38:47.419961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.419523 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:38:47.438896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.438854 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-token\" (UniqueName: \"kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-token\") pod \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " Oct 30 16:38:47.439119 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439048 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-data\") pod \"64654948-d621-4884-bb13-0501973a1d4f\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " Oct 30 16:38:47.439119 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439090 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64654948-d621-4884-bb13-0501973a1d4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64654948-d621-4884-bb13-0501973a1d4f-istio-podinfo\") pod \"64654948-d621-4884-bb13-0501973a1d4f\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " Oct 30 16:38:47.439235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439133 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-certs\") pod \"64654948-d621-4884-bb13-0501973a1d4f\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " Oct 30 16:38:47.439288 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439248 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-data\") pod \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " Oct 30 16:38:47.439338 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439292 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64654948-d621-4884-bb13-0501973a1d4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64654948-d621-4884-bb13-0501973a1d4f-istiod-ca-cert\") pod \"64654948-d621-4884-bb13-0501973a1d4f\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " Oct 30 16:38:47.439385 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439339 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-certs\") pod \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " Oct 30 16:38:47.439439 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439422 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-envoy\") pod \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " Oct 30 16:38:47.439494 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439464 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-envoy\") pod \"64654948-d621-4884-bb13-0501973a1d4f\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " Oct 30 16:38:47.439548 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439492 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-socket\") pod \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " Oct 30 16:38:47.439548 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439524 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-kube-api-access-2x76l\" (UniqueName: \"kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-kube-api-access-2x76l\") pod \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " Oct 30 16:38:47.439643 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439588 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istiod-ca-cert\") pod \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " Oct 30 16:38:47.439712 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439671 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-credential-socket\") pod \"64654948-d621-4884-bb13-0501973a1d4f\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " Oct 30 16:38:47.439712 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439699 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-podinfo\") pod \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " Oct 30 16:38:47.439806 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439747 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-istio-token\" (UniqueName: \"kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-istio-token\") pod \"64654948-d621-4884-bb13-0501973a1d4f\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " Oct 30 16:38:47.439888 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439869 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-credential-socket\") pod \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\" (UID: \"b2cafd8b-2615-423d-afd5-5a7fed76a7a6\") " Oct 30 16:38:47.439961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439938 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-kube-api-access-tgk2m\" (UniqueName: \"kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-kube-api-access-tgk2m\") pod \"64654948-d621-4884-bb13-0501973a1d4f\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " Oct 30 16:38:47.440034 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.439986 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-socket\") pod \"64654948-d621-4884-bb13-0501973a1d4f\" (UID: \"64654948-d621-4884-bb13-0501973a1d4f\") " Oct 30 16:38:47.440684 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.440481 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-credential-socket" pod "64654948-d621-4884-bb13-0501973a1d4f" (UID: "64654948-d621-4884-bb13-0501973a1d4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.440804 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.440707 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-socket" pod "64654948-d621-4884-bb13-0501973a1d4f" (UID: "64654948-d621-4884-bb13-0501973a1d4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.441357 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.441129 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.441357 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.441157 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.441829 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.441575 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-certs" pod "64654948-d621-4884-bb13-0501973a1d4f" (UID: "64654948-d621-4884-bb13-0501973a1d4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.442526 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.442501 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istiod-ca-cert" pod "b2cafd8b-2615-423d-afd5-5a7fed76a7a6" (UID: "b2cafd8b-2615-423d-afd5-5a7fed76a7a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:47.442619 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.442562 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-data" pod "b2cafd8b-2615-423d-afd5-5a7fed76a7a6" (UID: "b2cafd8b-2615-423d-afd5-5a7fed76a7a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.442619 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.442582 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-socket" pod "b2cafd8b-2615-423d-afd5-5a7fed76a7a6" (UID: "b2cafd8b-2615-423d-afd5-5a7fed76a7a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.443066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.443048 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-data" pod "64654948-d621-4884-bb13-0501973a1d4f" (UID: "64654948-d621-4884-bb13-0501973a1d4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.443436 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.443140 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64654948-d621-4884-bb13-0501973a1d4f-istiod-ca-cert" pod "64654948-d621-4884-bb13-0501973a1d4f" (UID: "64654948-d621-4884-bb13-0501973a1d4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:47.443436 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.443183 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-credential-socket" pod "b2cafd8b-2615-423d-afd5-5a7fed76a7a6" (UID: "b2cafd8b-2615-423d-afd5-5a7fed76a7a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.443436 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.443403 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-certs" pod "b2cafd8b-2615-423d-afd5-5a7fed76a7a6" (UID: "b2cafd8b-2615-423d-afd5-5a7fed76a7a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.516030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.515877 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64654948-d621-4884-bb13-0501973a1d4f-istio-podinfo" pod "64654948-d621-4884-bb13-0501973a1d4f" (UID: "64654948-d621-4884-bb13-0501973a1d4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:38:47.516030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.515958 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-podinfo" pod "b2cafd8b-2615-423d-afd5-5a7fed76a7a6" (UID: "b2cafd8b-2615-423d-afd5-5a7fed76a7a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:38:47.516649 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.516349 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-envoy" pod "64654948-d621-4884-bb13-0501973a1d4f" (UID: "64654948-d621-4884-bb13-0501973a1d4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.516649 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.516534 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-token" pod "b2cafd8b-2615-423d-afd5-5a7fed76a7a6" (UID: "b2cafd8b-2615-423d-afd5-5a7fed76a7a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:47.516649 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.516563 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-istio-token" pod "64654948-d621-4884-bb13-0501973a1d4f" (UID: "64654948-d621-4884-bb13-0501973a1d4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:47.516649 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.516629 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-kube-api-access-2x76l" pod "b2cafd8b-2615-423d-afd5-5a7fed76a7a6" (UID: "b2cafd8b-2615-423d-afd5-5a7fed76a7a6"). InnerVolumeSpecName "kube-api-access-2x76l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:47.516883 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.516761 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-envoy" pod "b2cafd8b-2615-423d-afd5-5a7fed76a7a6" (UID: "b2cafd8b-2615-423d-afd5-5a7fed76a7a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.517772 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.517731 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-kube-api-access-tgk2m" pod "64654948-d621-4884-bb13-0501973a1d4f" (UID: "64654948-d621-4884-bb13-0501973a1d4f"). InnerVolumeSpecName "kube-api-access-tgk2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:47.537348 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.537319 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:38:47.539284 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.539254 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:38:47.542653 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542623 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.542782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542655 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.542782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542672 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.542782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542688 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgk2m\" (UniqueName: \"kubernetes.io/projected/64654948-d621-4884-bb13-0501973a1d4f-kube-api-access-tgk2m\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.542782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542700 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.542782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542712 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.542782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542725 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64654948-d621-4884-bb13-0501973a1d4f-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.542782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542737 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.542782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542750 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.542782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542764 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64654948-d621-4884-bb13-0501973a1d4f-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.542782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542776 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.543340 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542788 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.543340 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542804 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64654948-d621-4884-bb13-0501973a1d4f-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.543340 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542830 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.543340 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542843 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x76l\" (UniqueName: \"kubernetes.io/projected/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-kube-api-access-2x76l\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.543340 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.542856 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cafd8b-2615-423d-afd5-5a7fed76a7a6-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.643871 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.643828 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-token\" (UniqueName: \"kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-token\") pod \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " Oct 30 16:38:47.644061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.643920 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-socket\") pod \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " Oct 30 16:38:47.644061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.643987 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-data\") pod \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " Oct 30 16:38:47.644061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644027 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istiod-ca-cert\") pod \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " Oct 30 16:38:47.644223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644096 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-kube-api-access-zv5bf\" (UniqueName: \"kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-kube-api-access-zv5bf\") pod \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " Oct 30 16:38:47.644223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644159 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-podinfo\") pod \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " Oct 30 16:38:47.644223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644186 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istiod-ca-cert\") pod \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " Oct 30 16:38:47.644368 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644259 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-certs\") pod \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " Oct 30 16:38:47.644368 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644293 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-credential-socket\") pod \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " Oct 30 16:38:47.644368 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644333 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-socket\") pod \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " Oct 30 16:38:47.644509 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644386 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-kube-api-access-zkphj\" (UniqueName: \"kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-kube-api-access-zkphj\") pod \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " Oct 30 16:38:47.644509 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644416 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-envoy\") pod \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " Oct 30 16:38:47.644509 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644446 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-credential-socket\") pod \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " Oct 30 16:38:47.644509 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644488 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-podinfo\") pod \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " Oct 30 16:38:47.644718 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644568 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-certs\") pod \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " Oct 30 16:38:47.644718 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644690 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-token\" (UniqueName: \"kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-token\") pod \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\" (UID: \"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46\") " Oct 30 16:38:47.644813 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644763 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-data\") pod \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " Oct 30 16:38:47.644858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.644812 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-envoy\") pod \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\" (UID: \"792c09a1-9d64-470a-8bd8-b2e1cef34b7a\") " Oct 30 16:38:47.646082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.645559 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-data" pod "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" (UID: "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.646082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.645762 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-socket" pod "792c09a1-9d64-470a-8bd8-b2e1cef34b7a" (UID: "792c09a1-9d64-470a-8bd8-b2e1cef34b7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.646082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.645808 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-socket" pod "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" (UID: "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.646082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.646067 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-certs" pod "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" (UID: "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.648575 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.647639 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-token" pod "792c09a1-9d64-470a-8bd8-b2e1cef34b7a" (UID: "792c09a1-9d64-470a-8bd8-b2e1cef34b7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:47.648575 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.647809 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-credential-socket" pod "792c09a1-9d64-470a-8bd8-b2e1cef34b7a" (UID: "792c09a1-9d64-470a-8bd8-b2e1cef34b7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.648575 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.647813 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-kube-api-access-zv5bf" pod "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" (UID: "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46"). InnerVolumeSpecName "kube-api-access-zv5bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:47.648575 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.647895 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istiod-ca-cert" pod "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" (UID: "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:47.648575 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.648008 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-data" pod "792c09a1-9d64-470a-8bd8-b2e1cef34b7a" (UID: "792c09a1-9d64-470a-8bd8-b2e1cef34b7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.648575 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.648299 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-certs" pod "792c09a1-9d64-470a-8bd8-b2e1cef34b7a" (UID: "792c09a1-9d64-470a-8bd8-b2e1cef34b7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.649085 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.649039 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istiod-ca-cert" pod "792c09a1-9d64-470a-8bd8-b2e1cef34b7a" (UID: "792c09a1-9d64-470a-8bd8-b2e1cef34b7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:47.649207 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.649085 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-credential-socket" pod "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" (UID: "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.651622 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.651415 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-kube-api-access-zkphj" pod "792c09a1-9d64-470a-8bd8-b2e1cef34b7a" (UID: "792c09a1-9d64-470a-8bd8-b2e1cef34b7a"). InnerVolumeSpecName "kube-api-access-zkphj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:47.651622 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.651501 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-podinfo" pod "792c09a1-9d64-470a-8bd8-b2e1cef34b7a" (UID: "792c09a1-9d64-470a-8bd8-b2e1cef34b7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:38:47.651622 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.651595 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-podinfo" pod "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" (UID: "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:38:47.652108 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.652072 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-token" pod "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" (UID: "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:47.652653 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.652631 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-envoy" pod "792c09a1-9d64-470a-8bd8-b2e1cef34b7a" (UID: "792c09a1-9d64-470a-8bd8-b2e1cef34b7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.652871 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.652850 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-envoy" pod "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" (UID: "fc6b5808-eb84-4f36-8d25-0c5b19ab3c46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:47.747009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.746955 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747006 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747017 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747026 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747035 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747043 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747052 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747061 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv5bf\" (UniqueName: \"kubernetes.io/projected/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-kube-api-access-zv5bf\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747070 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747078 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747085 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747094 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747102 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747110 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkphj\" (UniqueName: \"kubernetes.io/projected/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-kube-api-access-zkphj\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747119 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747127 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/792c09a1-9d64-470a-8bd8-b2e1cef34b7a-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747135 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.747225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.747144 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:47.886281 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.878870 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" Oct 30 16:38:47.886281 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.880402 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" event={"ID":"64654948-d621-4884-bb13-0501973a1d4f","Type":"ContainerDied","Data":"2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a"} Oct 30 16:38:47.886281 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.880496 5838 scope.go:122] "RemoveContainer" containerID="2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a" Oct 30 16:38:47.886281 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.880659 5838 generic.go:356] "Generic (PLEG): container finished" podID="64654948-d621-4884-bb13-0501973a1d4f" containerID="2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a" exitCode=0 Oct 30 16:38:47.886281 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.880737 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p" event={"ID":"64654948-d621-4884-bb13-0501973a1d4f","Type":"ContainerDied","Data":"d576f688427221b093498c3785cdf4748f372d20b89daafcb270fda69a6d29a8"} Oct 30 16:38:47.906508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.888914 5838 generic.go:356] "Generic (PLEG): container finished" podID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerID="f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a" exitCode=0 Oct 30 16:38:47.906508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.889093 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" event={"ID":"792c09a1-9d64-470a-8bd8-b2e1cef34b7a","Type":"ContainerDied","Data":"f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a"} Oct 30 16:38:47.906508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.889153 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" event={"ID":"792c09a1-9d64-470a-8bd8-b2e1cef34b7a","Type":"ContainerDied","Data":"7cd8654f5e9b0a1d483f6c5f1fb27e6156b68e2a5a865f8cf71b8610ffcf9b8f"} Oct 30 16:38:47.906508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.889237 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4" Oct 30 16:38:47.921585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.912048 5838 generic.go:356] "Generic (PLEG): container finished" podID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerID="e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced" exitCode=0 Oct 30 16:38:47.921585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.912203 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" event={"ID":"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46","Type":"ContainerDied","Data":"e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced"} Oct 30 16:38:47.921585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.912265 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" event={"ID":"fc6b5808-eb84-4f36-8d25-0c5b19ab3c46","Type":"ContainerDied","Data":"8eb37cc693329394ae8d9a6a5e2bb152abdd6f274c9a96be4a514fafa9b642e5"} Oct 30 16:38:47.921585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.912371 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg" Oct 30 16:38:47.930413 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.929338 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" event={"ID":"555c9e26-cfa2-4413-81a5-f21a77718534","Type":"ContainerStarted","Data":"2659d0dcebb744aa43882cb560ec9b2ac67eceb563a24959bba23f768683bc8a"} Oct 30 16:38:47.930413 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.929414 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" event={"ID":"555c9e26-cfa2-4413-81a5-f21a77718534","Type":"ContainerStarted","Data":"040fefd029105586caa1cfe99e8bf6bfe5311c008dc0923de5fce00aa0a47fab"} Oct 30 16:38:47.951004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.947118 5838 generic.go:356] "Generic (PLEG): container finished" podID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" containerID="8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c" exitCode=0 Oct 30 16:38:47.951004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.947646 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" Oct 30 16:38:47.952116 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.951307 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" event={"ID":"b2cafd8b-2615-423d-afd5-5a7fed76a7a6","Type":"ContainerDied","Data":"8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c"} Oct 30 16:38:47.952116 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.951379 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn" event={"ID":"b2cafd8b-2615-423d-afd5-5a7fed76a7a6","Type":"ContainerDied","Data":"c4903f18303d081e6dc50c2ac4a2c854850213f43b4c7b1266008455bcdcfaa0"} Oct 30 16:38:47.962920 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.962901 5838 scope.go:122] "RemoveContainer" containerID="2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a" Oct 30 16:38:47.963467 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:47.963440 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a\": container with ID starting with 2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a not found: ID does not exist" containerID="2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a" Oct 30 16:38:47.963551 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.963479 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a"} err="failed to get container status \"2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a\": rpc error: code = NotFound desc = could not find container \"2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a\": container with ID starting with 2012ea74d9eea730292ea8e8f51a3130d198dfafeeefcb2061630d8602a58f5a not found: ID does not exist" Oct 30 16:38:47.963551 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.963500 5838 scope.go:122] "RemoveContainer" containerID="f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a" Oct 30 16:38:47.977242 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.976998 5838 scope.go:122] "RemoveContainer" containerID="f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a" Oct 30 16:38:47.977362 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:47.977337 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a\": container with ID starting with f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a not found: ID does not exist" containerID="f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a" Oct 30 16:38:47.977426 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.977372 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a"} err="failed to get container status \"f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a\": rpc error: code = NotFound desc = could not find container \"f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a\": container with ID starting with f891bc0cec16833dc00192e3b658d722c0d68631671eb29006a9ed9bc1bc543a not found: ID does not exist" Oct 30 16:38:47.977426 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.977393 5838 scope.go:122] "RemoveContainer" containerID="e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced" Oct 30 16:38:47.986946 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.986894 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" podStartSLOduration=1.98687879 podStartE2EDuration="1.98687879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:47.981577136 +0000 UTC m=+1089.730166542" watchObservedRunningTime="2026-10-30 16:38:47.98687879 +0000 UTC m=+1089.735468177" Oct 30 16:38:47.990606 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.990581 5838 scope.go:122] "RemoveContainer" containerID="e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced" Oct 30 16:38:47.990964 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:47.990943 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced\": container with ID starting with e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced not found: ID does not exist" containerID="e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced" Oct 30 16:38:47.990964 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.990985 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced"} err="failed to get container status \"e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced\": rpc error: code = NotFound desc = could not find container \"e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced\": container with ID starting with e304c30df2a1d10f1c089310e4f66b5ae114e59fceb93497783e5c77711d3ced not found: ID does not exist" Oct 30 16:38:47.991149 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:47.991010 5838 scope.go:122] "RemoveContainer" containerID="8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c" Oct 30 16:38:48.002961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.002937 5838 scope.go:122] "RemoveContainer" containerID="8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c" Oct 30 16:38:48.003318 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:48.003288 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c\": container with ID starting with 8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c not found: ID does not exist" containerID="8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c" Oct 30 16:38:48.003409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.003328 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c"} err="failed to get container status \"8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c\": rpc error: code = NotFound desc = could not find container \"8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c\": container with ID starting with 8412305377a595bf706dcc1177949cc756d6f2c744e3e674a25a65759043c95c not found: ID does not exist" Oct 30 16:38:48.023642 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.023602 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4"] Oct 30 16:38:48.029454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.029424 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-92bv4"] Oct 30 16:38:48.085155 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.085123 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg"] Oct 30 16:38:48.093254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.093218 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-gktrg"] Oct 30 16:38:48.121507 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.121430 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p"] Oct 30 16:38:48.126880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.126853 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-5x95p"] Oct 30 16:38:48.157528 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.157499 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:48.158516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.158489 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn"] Oct 30 16:38:48.163984 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.163945 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-063dcc-openshift-default-69f66fb99-t9xhn"] Oct 30 16:38:48.272401 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.272371 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-openshift-ca\") pod \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\" (UID: \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\") " Oct 30 16:38:48.272568 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.272552 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-xks-ca\") pod \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\" (UID: \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\") " Oct 30 16:38:48.272640 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.272627 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-kube-api-access-m9j7k\" (UniqueName: \"kubernetes.io/projected/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-kube-api-access-m9j7k\") pod \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\" (UID: \"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9\") " Oct 30 16:38:48.272733 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.272710 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-openshift-ca" pod "f9bc4a05-8c71-4073-a2c3-779b80a3a3d9" (UID: "f9bc4a05-8c71-4073-a2c3-779b80a3a3d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:48.273329 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.273301 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:48.274788 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.274762 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-xks-ca" pod "f9bc4a05-8c71-4073-a2c3-779b80a3a3d9" (UID: "f9bc4a05-8c71-4073-a2c3-779b80a3a3d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:48.274889 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.274837 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-kube-api-access-m9j7k" pod "f9bc4a05-8c71-4073-a2c3-779b80a3a3d9" (UID: "f9bc4a05-8c71-4073-a2c3-779b80a3a3d9"). InnerVolumeSpecName "kube-api-access-m9j7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:48.374353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.374282 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:48.374353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.374309 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9j7k\" (UniqueName: \"kubernetes.io/projected/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9-kube-api-access-m9j7k\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:48.399008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.398961 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64654948-d621-4884-bb13-0501973a1d4f" path="/var/lib/kubelet/pods/64654948-d621-4884-bb13-0501973a1d4f/volumes" Oct 30 16:38:48.399471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.399457 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" path="/var/lib/kubelet/pods/792c09a1-9d64-470a-8bd8-b2e1cef34b7a/volumes" Oct 30 16:38:48.399904 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.399891 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" path="/var/lib/kubelet/pods/b2cafd8b-2615-423d-afd5-5a7fed76a7a6/volumes" Oct 30 16:38:48.400613 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.400594 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" path="/var/lib/kubelet/pods/fc6b5808-eb84-4f36-8d25-0c5b19ab3c46/volumes" Oct 30 16:38:48.902274 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.902241 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" podUID="0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:38:48.902637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.902241 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" podUID="a1a40be2-e7d1-49f9-a304-2623278732ba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:38:48.957952 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.957921 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" Oct 30 16:38:48.958125 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.957926 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd" event={"ID":"f9bc4a05-8c71-4073-a2c3-779b80a3a3d9","Type":"ContainerDied","Data":"77699c53e74ddde06ff41d7f84705390e1e31ec2702889f4012d03a6c9cc0352"} Oct 30 16:38:48.958125 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.958045 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77699c53e74ddde06ff41d7f84705390e1e31ec2702889f4012d03a6c9cc0352" Oct 30 16:38:48.963068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.963035 5838 generic.go:356] "Generic (PLEG): container finished" podID="555c9e26-cfa2-4413-81a5-f21a77718534" containerID="2659d0dcebb744aa43882cb560ec9b2ac67eceb563a24959bba23f768683bc8a" exitCode=0 Oct 30 16:38:48.963212 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:48.963110 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" event={"ID":"555c9e26-cfa2-4413-81a5-f21a77718534","Type":"ContainerDied","Data":"2659d0dcebb744aa43882cb560ec9b2ac67eceb563a24959bba23f768683bc8a"} Oct 30 16:38:50.221840 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:50.221806 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:50.314390 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:50.314287 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-xks-ca\") pod \"555c9e26-cfa2-4413-81a5-f21a77718534\" (UID: \"555c9e26-cfa2-4413-81a5-f21a77718534\") " Oct 30 16:38:50.314390 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:50.314361 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-openshift-ca\") pod \"555c9e26-cfa2-4413-81a5-f21a77718534\" (UID: \"555c9e26-cfa2-4413-81a5-f21a77718534\") " Oct 30 16:38:50.314638 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:50.314577 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555c9e26-cfa2-4413-81a5-f21a77718534-kube-api-access-vw84j\" (UniqueName: \"kubernetes.io/projected/555c9e26-cfa2-4413-81a5-f21a77718534-kube-api-access-vw84j\") pod \"555c9e26-cfa2-4413-81a5-f21a77718534\" (UID: \"555c9e26-cfa2-4413-81a5-f21a77718534\") " Oct 30 16:38:50.315091 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:50.314951 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-openshift-ca" pod "555c9e26-cfa2-4413-81a5-f21a77718534" (UID: "555c9e26-cfa2-4413-81a5-f21a77718534"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:50.317153 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:50.317131 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555c9e26-cfa2-4413-81a5-f21a77718534-kube-api-access-vw84j" pod "555c9e26-cfa2-4413-81a5-f21a77718534" (UID: "555c9e26-cfa2-4413-81a5-f21a77718534"). InnerVolumeSpecName "kube-api-access-vw84j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:50.317262 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:50.317131 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-xks-ca" pod "555c9e26-cfa2-4413-81a5-f21a77718534" (UID: "555c9e26-cfa2-4413-81a5-f21a77718534"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:50.417283 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:50.417255 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw84j\" (UniqueName: \"kubernetes.io/projected/555c9e26-cfa2-4413-81a5-f21a77718534-kube-api-access-vw84j\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:50.417465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:50.417441 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:50.417557 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:50.417470 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/555c9e26-cfa2-4413-81a5-f21a77718534-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.039414 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.039386 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" Oct 30 16:38:51.039569 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.039428 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg" event={"ID":"555c9e26-cfa2-4413-81a5-f21a77718534","Type":"ContainerDied","Data":"040fefd029105586caa1cfe99e8bf6bfe5311c008dc0923de5fce00aa0a47fab"} Oct 30 16:38:51.039569 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.039478 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="040fefd029105586caa1cfe99e8bf6bfe5311c008dc0923de5fce00aa0a47fab" Oct 30 16:38:51.378605 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.378573 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:38:51.426400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.426370 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:38:51.444131 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.444104 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-podinfo\") pod \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " Oct 30 16:38:51.444285 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.444267 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-certs\") pod \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " Oct 30 16:38:51.444601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.444303 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-socket\") pod \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " Oct 30 16:38:51.444601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.444350 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-token\" (UniqueName: \"kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-token\") pod \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " Oct 30 16:38:51.444601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.444488 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-envoy\") pod \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " Oct 30 16:38:51.444601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.444533 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-kube-api-access-t6zgt\" (UniqueName: \"kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-kube-api-access-t6zgt\") pod \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " Oct 30 16:38:51.444601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.444572 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istiod-ca-cert\") pod \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " Oct 30 16:38:51.444845 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.444605 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-credential-socket\") pod \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " Oct 30 16:38:51.444845 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.444728 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-data\") pod \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\" (UID: \"14848f01-2514-46ed-b1e3-cfbbca9ab49d\") " Oct 30 16:38:51.445735 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.445231 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-socket" pod "14848f01-2514-46ed-b1e3-cfbbca9ab49d" (UID: "14848f01-2514-46ed-b1e3-cfbbca9ab49d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.445735 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.445307 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-certs" pod "14848f01-2514-46ed-b1e3-cfbbca9ab49d" (UID: "14848f01-2514-46ed-b1e3-cfbbca9ab49d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.445735 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.445457 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-credential-socket" pod "14848f01-2514-46ed-b1e3-cfbbca9ab49d" (UID: "14848f01-2514-46ed-b1e3-cfbbca9ab49d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.445735 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.445478 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istiod-ca-cert" pod "14848f01-2514-46ed-b1e3-cfbbca9ab49d" (UID: "14848f01-2514-46ed-b1e3-cfbbca9ab49d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:51.445735 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.445598 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.445735 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.445616 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.445735 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.445631 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.446204 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.445857 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-data" pod "14848f01-2514-46ed-b1e3-cfbbca9ab49d" (UID: "14848f01-2514-46ed-b1e3-cfbbca9ab49d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.448368 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.448220 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-kube-api-access-t6zgt" pod "14848f01-2514-46ed-b1e3-cfbbca9ab49d" (UID: "14848f01-2514-46ed-b1e3-cfbbca9ab49d"). InnerVolumeSpecName "kube-api-access-t6zgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:51.448588 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.448551 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-token" pod "14848f01-2514-46ed-b1e3-cfbbca9ab49d" (UID: "14848f01-2514-46ed-b1e3-cfbbca9ab49d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:51.449869 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.449842 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-podinfo" pod "14848f01-2514-46ed-b1e3-cfbbca9ab49d" (UID: "14848f01-2514-46ed-b1e3-cfbbca9ab49d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:38:51.451223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.451203 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-envoy" pod "14848f01-2514-46ed-b1e3-cfbbca9ab49d" (UID: "14848f01-2514-46ed-b1e3-cfbbca9ab49d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.473416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.473389 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:38:51.546818 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.546784 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-socket\") pod \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " Oct 30 16:38:51.546818 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.546834 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-socket\") pod \"d212d906-4b33-44b1-9079-1b5ae0000f3b\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " Oct 30 16:38:51.547118 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.546872 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-kube-api-access-hz9cl\" (UniqueName: \"kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-kube-api-access-hz9cl\") pod \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " Oct 30 16:38:51.547118 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.546905 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-podinfo\") pod \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " Oct 30 16:38:51.547118 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.546939 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istiod-ca-cert\") pod \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " Oct 30 16:38:51.547118 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.546960 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-data\") pod \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " Oct 30 16:38:51.547118 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.547099 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-credential-socket\") pod \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " Oct 30 16:38:51.547300 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.547139 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-certs\") pod \"d212d906-4b33-44b1-9079-1b5ae0000f3b\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " Oct 30 16:38:51.547300 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.547197 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-token\" (UniqueName: \"kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-token\") pod \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " Oct 30 16:38:51.547300 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.547269 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-envoy\") pod \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " Oct 30 16:38:51.547429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.547304 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-podinfo\") pod \"d212d906-4b33-44b1-9079-1b5ae0000f3b\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " Oct 30 16:38:51.547429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.547355 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-data\") pod \"d212d906-4b33-44b1-9079-1b5ae0000f3b\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " Oct 30 16:38:51.547429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.547383 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-certs\") pod \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\" (UID: \"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf\") " Oct 30 16:38:51.547563 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.547441 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-kube-api-access-lcg7r\" (UniqueName: \"kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-kube-api-access-lcg7r\") pod \"d212d906-4b33-44b1-9079-1b5ae0000f3b\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " Oct 30 16:38:51.547608 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.547558 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-token\" (UniqueName: \"kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-token\") pod \"d212d906-4b33-44b1-9079-1b5ae0000f3b\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " Oct 30 16:38:51.548057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.547665 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d212d906-4b33-44b1-9079-1b5ae0000f3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d212d906-4b33-44b1-9079-1b5ae0000f3b-istiod-ca-cert\") pod \"d212d906-4b33-44b1-9079-1b5ae0000f3b\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " Oct 30 16:38:51.548057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.547721 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-credential-socket\") pod \"d212d906-4b33-44b1-9079-1b5ae0000f3b\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " Oct 30 16:38:51.548057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.547783 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-envoy\") pod \"d212d906-4b33-44b1-9079-1b5ae0000f3b\" (UID: \"d212d906-4b33-44b1-9079-1b5ae0000f3b\") " Oct 30 16:38:51.551016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.549102 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.551016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.549133 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.551016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.549165 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6zgt\" (UniqueName: \"kubernetes.io/projected/14848f01-2514-46ed-b1e3-cfbbca9ab49d-kube-api-access-t6zgt\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.551016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.549181 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.551016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.549195 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.551016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.549185 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-credential-socket" pod "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" (UID: "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.551016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.549207 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14848f01-2514-46ed-b1e3-cfbbca9ab49d-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.551016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.550110 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istiod-ca-cert" pod "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" (UID: "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:51.551016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.550429 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-certs" pod "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" (UID: "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.551016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.550824 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-socket" pod "d212d906-4b33-44b1-9079-1b5ae0000f3b" (UID: "d212d906-4b33-44b1-9079-1b5ae0000f3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.551016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.550927 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-socket" pod "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" (UID: "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.551820 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.551794 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-data" pod "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" (UID: "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.552870 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.552613 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d212d906-4b33-44b1-9079-1b5ae0000f3b-istiod-ca-cert" pod "d212d906-4b33-44b1-9079-1b5ae0000f3b" (UID: "d212d906-4b33-44b1-9079-1b5ae0000f3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:51.552870 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.552756 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-token" pod "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" (UID: "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:51.553197 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.552990 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-data" pod "d212d906-4b33-44b1-9079-1b5ae0000f3b" (UID: "d212d906-4b33-44b1-9079-1b5ae0000f3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.553197 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.553181 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-credential-socket" pod "d212d906-4b33-44b1-9079-1b5ae0000f3b" (UID: "d212d906-4b33-44b1-9079-1b5ae0000f3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.553528 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.553506 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-certs" pod "d212d906-4b33-44b1-9079-1b5ae0000f3b" (UID: "d212d906-4b33-44b1-9079-1b5ae0000f3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.554533 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.554479 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-podinfo" pod "d212d906-4b33-44b1-9079-1b5ae0000f3b" (UID: "d212d906-4b33-44b1-9079-1b5ae0000f3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:38:51.554764 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.554722 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-podinfo" pod "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" (UID: "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:38:51.555594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.555566 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-kube-api-access-lcg7r" pod "d212d906-4b33-44b1-9079-1b5ae0000f3b" (UID: "d212d906-4b33-44b1-9079-1b5ae0000f3b"). InnerVolumeSpecName "kube-api-access-lcg7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:51.556480 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.556440 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-token" pod "d212d906-4b33-44b1-9079-1b5ae0000f3b" (UID: "d212d906-4b33-44b1-9079-1b5ae0000f3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:51.556724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.556696 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-envoy" pod "d212d906-4b33-44b1-9079-1b5ae0000f3b" (UID: "d212d906-4b33-44b1-9079-1b5ae0000f3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.557085 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.557063 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-kube-api-access-hz9cl" pod "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" (UID: "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf"). InnerVolumeSpecName "kube-api-access-hz9cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:51.557160 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.557063 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-envoy" pod "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" (UID: "2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:38:51.650811 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650729 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.650811 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650769 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.650811 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650785 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.650811 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650797 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.650811 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650813 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650827 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcg7r\" (UniqueName: \"kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-kube-api-access-lcg7r\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650840 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650853 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d212d906-4b33-44b1-9079-1b5ae0000f3b-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650865 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650877 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650894 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650910 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650927 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz9cl\" (UniqueName: \"kubernetes.io/projected/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-kube-api-access-hz9cl\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650939 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650951 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650964 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.650997 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:51.651093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:51.651009 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d212d906-4b33-44b1-9079-1b5ae0000f3b-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:52.054073 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.054020 5838 generic.go:356] "Generic (PLEG): container finished" podID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerID="810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191" exitCode=0 Oct 30 16:38:52.054247 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.054068 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" event={"ID":"14848f01-2514-46ed-b1e3-cfbbca9ab49d","Type":"ContainerDied","Data":"810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191"} Oct 30 16:38:52.054247 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.054042 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" Oct 30 16:38:52.054247 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.054138 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch" event={"ID":"14848f01-2514-46ed-b1e3-cfbbca9ab49d","Type":"ContainerDied","Data":"3cfa0bd4a52f4e2be459fa619d893e41dc3e0fbb1c042d8ff179366c444213f4"} Oct 30 16:38:52.054247 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.054166 5838 scope.go:122] "RemoveContainer" containerID="810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191" Oct 30 16:38:52.056345 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.056318 5838 generic.go:356] "Generic (PLEG): container finished" podID="d212d906-4b33-44b1-9079-1b5ae0000f3b" containerID="aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96" exitCode=0 Oct 30 16:38:52.056469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.056391 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" Oct 30 16:38:52.056469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.056409 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" event={"ID":"d212d906-4b33-44b1-9079-1b5ae0000f3b","Type":"ContainerDied","Data":"aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96"} Oct 30 16:38:52.056469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.056463 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr" event={"ID":"d212d906-4b33-44b1-9079-1b5ae0000f3b","Type":"ContainerDied","Data":"127ad2f939f5b3de988e220e367d44f96343350a04ce978db9e184e92d52979c"} Oct 30 16:38:52.060663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.060635 5838 generic.go:356] "Generic (PLEG): container finished" podID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerID="84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213" exitCode=0 Oct 30 16:38:52.060768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.060682 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" event={"ID":"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf","Type":"ContainerDied","Data":"84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213"} Oct 30 16:38:52.060768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.060726 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" event={"ID":"2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf","Type":"ContainerDied","Data":"c531686b06b4e62c5547593e52c8ad6254f3573c3519fe7b402057a8c34d9aac"} Oct 30 16:38:52.060948 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.060923 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw" Oct 30 16:38:52.069333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.069309 5838 scope.go:122] "RemoveContainer" containerID="810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191" Oct 30 16:38:52.069642 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:52.069624 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191\": container with ID starting with 810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191 not found: ID does not exist" containerID="810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191" Oct 30 16:38:52.069700 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.069655 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191"} err="failed to get container status \"810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191\": rpc error: code = NotFound desc = could not find container \"810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191\": container with ID starting with 810a426b6d7e259d5681c8dcd0dd1968f6b24f0c0cc90f42ffb5b82a46e1b191 not found: ID does not exist" Oct 30 16:38:52.069700 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.069683 5838 scope.go:122] "RemoveContainer" containerID="aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96" Oct 30 16:38:52.081124 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.081103 5838 scope.go:122] "RemoveContainer" containerID="aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96" Oct 30 16:38:52.081426 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:52.081407 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96\": container with ID starting with aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96 not found: ID does not exist" containerID="aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96" Oct 30 16:38:52.081511 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.081438 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96"} err="failed to get container status \"aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96\": rpc error: code = NotFound desc = could not find container \"aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96\": container with ID starting with aeba91315742cdb33a02f7324672e00db24860924a0fb6a7d7d391a2fd4a7a96 not found: ID does not exist" Oct 30 16:38:52.081511 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.081463 5838 scope.go:122] "RemoveContainer" containerID="84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213" Oct 30 16:38:52.092765 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.092745 5838 scope.go:122] "RemoveContainer" containerID="84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213" Oct 30 16:38:52.093108 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:38:52.093088 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213\": container with ID starting with 84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213 not found: ID does not exist" containerID="84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213" Oct 30 16:38:52.093167 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.093116 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213"} err="failed to get container status \"84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213\": rpc error: code = NotFound desc = could not find container \"84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213\": container with ID starting with 84401c44b0e504faddd44639e1ef09297097b154ca1dfdf8ef4d9329b428f213 not found: ID does not exist" Oct 30 16:38:52.122149 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.122112 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch"] Oct 30 16:38:52.127409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.127383 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-vbvch"] Oct 30 16:38:52.163348 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.163310 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr"] Oct 30 16:38:52.167619 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.167588 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-nqdzr"] Oct 30 16:38:52.201048 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.201013 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw"] Oct 30 16:38:52.208471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.208438 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3e4f91-openshift-default-7fb6b55b45-xnqgw"] Oct 30 16:38:52.397892 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.397856 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" path="/var/lib/kubelet/pods/14848f01-2514-46ed-b1e3-cfbbca9ab49d/volumes" Oct 30 16:38:52.398431 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.398416 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" path="/var/lib/kubelet/pods/2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf/volumes" Oct 30 16:38:52.398868 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:52.398855 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d212d906-4b33-44b1-9079-1b5ae0000f3b" path="/var/lib/kubelet/pods/d212d906-4b33-44b1-9079-1b5ae0000f3b/volumes" Oct 30 16:38:55.628062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.628022 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4"] Oct 30 16:38:55.638143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.638107 5838 cpu_manager.go:414] "containerMap: removing container" podUID="64654948-d621-4884-bb13-0501973a1d4f" containerName="istio-proxy" Oct 30 16:38:55.638367 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.638349 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64654948-d621-4884-bb13-0501973a1d4f" containerName="istio-proxy" Oct 30 16:38:55.638493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.638482 5838 cpu_manager.go:414] "containerMap: removing container" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" Oct 30 16:38:55.638568 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.638557 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" Oct 30 16:38:55.638653 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.638643 5838 cpu_manager.go:414] "containerMap: removing container" podUID="f9bc4a05-8c71-4073-a2c3-779b80a3a3d9" containerName="revoke-keys" Oct 30 16:38:55.638724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.638714 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bc4a05-8c71-4073-a2c3-779b80a3a3d9" containerName="revoke-keys" Oct 30 16:38:55.638801 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.638792 5838 cpu_manager.go:414] "containerMap: removing container" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" Oct 30 16:38:55.638871 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.638862 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" Oct 30 16:38:55.638950 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.638941 5838 cpu_manager.go:414] "containerMap: removing container" podUID="555c9e26-cfa2-4413-81a5-f21a77718534" containerName="revoke-keys" Oct 30 16:38:55.639054 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.639039 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555c9e26-cfa2-4413-81a5-f21a77718534" containerName="revoke-keys" Oct 30 16:38:55.639178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.639167 5838 cpu_manager.go:414] "containerMap: removing container" podUID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" containerName="istio-proxy" Oct 30 16:38:55.639252 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.639242 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" containerName="istio-proxy" Oct 30 16:38:55.644045 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.643955 5838 cpu_manager.go:414] "containerMap: removing container" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerName="istio-proxy" Oct 30 16:38:55.644045 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644046 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerName="istio-proxy" Oct 30 16:38:55.644278 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644065 5838 cpu_manager.go:414] "containerMap: removing container" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" Oct 30 16:38:55.644278 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644073 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" Oct 30 16:38:55.644278 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644103 5838 cpu_manager.go:414] "containerMap: removing container" podUID="d212d906-4b33-44b1-9079-1b5ae0000f3b" containerName="istio-proxy" Oct 30 16:38:55.644278 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644110 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d212d906-4b33-44b1-9079-1b5ae0000f3b" containerName="istio-proxy" Oct 30 16:38:55.644484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644335 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="555c9e26-cfa2-4413-81a5-f21a77718534" containerName="revoke-keys" Oct 30 16:38:55.644484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644357 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2cafd8b-2615-423d-afd5-5a7fed76a7a6" containerName="istio-proxy" Oct 30 16:38:55.644484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644369 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="14848f01-2514-46ed-b1e3-cfbbca9ab49d" containerName="istio-proxy" Oct 30 16:38:55.644484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644377 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc6b5808-eb84-4f36-8d25-0c5b19ab3c46" containerName="istio-proxy" Oct 30 16:38:55.644484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644390 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d212d906-4b33-44b1-9079-1b5ae0000f3b" containerName="istio-proxy" Oct 30 16:38:55.644484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644404 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="64654948-d621-4884-bb13-0501973a1d4f" containerName="istio-proxy" Oct 30 16:38:55.644484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644416 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2bb0a9-1ccd-4187-bb7a-ffe7871527bf" containerName="istio-proxy" Oct 30 16:38:55.644484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644425 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bc4a05-8c71-4073-a2c3-779b80a3a3d9" containerName="revoke-keys" Oct 30 16:38:55.644484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.644435 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="792c09a1-9d64-470a-8bd8-b2e1cef34b7a" containerName="istio-proxy" Oct 30 16:38:55.652913 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.652880 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:38:55.657811 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.657778 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4"] Oct 30 16:38:55.678823 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.678796 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:38:55.678823 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.678803 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:55.755719 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.755670 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4\" (UID: \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:38:55.755930 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.755859 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4\" (UID: \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:38:55.756068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.755964 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgnl6\" (UniqueName: \"kubernetes.io/projected/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-kube-api-access-fgnl6\") pod \"maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4\" (UID: \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:38:55.856933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.856891 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4\" (UID: \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:38:55.857145 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.857059 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4\" (UID: \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:38:55.857145 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.857115 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgnl6\" (UniqueName: \"kubernetes.io/projected/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-kube-api-access-fgnl6\") pod \"maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4\" (UID: \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:38:55.857726 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.857704 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4\" (UID: \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:38:55.859442 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.859416 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4\" (UID: \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:38:55.881391 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.881324 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgnl6\" (UniqueName: \"kubernetes.io/projected/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-kube-api-access-fgnl6\") pod \"maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4\" (UID: \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:38:55.972713 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:55.972668 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:38:56.280112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:56.280068 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4"] Oct 30 16:38:56.281350 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:38:56.281320 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5742bf_5c5a_4054_8b2a_cb4b379cbaf5.slice/crio-a2a21bff88e043f7ba7a88f3ee2e438966d7a193171f1dafa8af3731c66d4f32 WatchSource:0}: Error finding container a2a21bff88e043f7ba7a88f3ee2e438966d7a193171f1dafa8af3731c66d4f32: Status 404 returned error can't find the container with id a2a21bff88e043f7ba7a88f3ee2e438966d7a193171f1dafa8af3731c66d4f32 Oct 30 16:38:57.018868 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.018835 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk"] Oct 30 16:38:57.023649 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.023626 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:38:57.058259 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.058216 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk"] Oct 30 16:38:57.072084 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.072047 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f3ff2b5-c9zdk\" (UID: \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:38:57.072325 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.072306 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f3ff2b5-c9zdk\" (UID: \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:38:57.072396 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.072369 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k84mx\" (UniqueName: \"kubernetes.io/projected/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-kube-api-access-k84mx\") pod \"maas-api-revoke-sub-1f3ff2b5-c9zdk\" (UID: \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:38:57.127680 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.127638 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" event={"ID":"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5","Type":"ContainerStarted","Data":"1caa717c263423c75e99c675b43bc367086791b0b629ca11ff10bfd61f59b555"} Oct 30 16:38:57.127956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.127931 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" event={"ID":"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5","Type":"ContainerStarted","Data":"a2a21bff88e043f7ba7a88f3ee2e438966d7a193171f1dafa8af3731c66d4f32"} Oct 30 16:38:57.169628 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.169557 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" podStartSLOduration=2.169544355 podStartE2EDuration="2.169544355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:57.167540214 +0000 UTC m=+1098.916129602" watchObservedRunningTime="2026-10-30 16:38:57.169544355 +0000 UTC m=+1098.918133741" Oct 30 16:38:57.174613 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.174576 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f3ff2b5-c9zdk\" (UID: \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:38:57.174850 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.174658 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k84mx\" (UniqueName: \"kubernetes.io/projected/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-kube-api-access-k84mx\") pod \"maas-api-revoke-sub-1f3ff2b5-c9zdk\" (UID: \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:38:57.174850 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.174704 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f3ff2b5-c9zdk\" (UID: \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:38:57.175326 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.175303 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f3ff2b5-c9zdk\" (UID: \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:38:57.177198 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.177179 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f3ff2b5-c9zdk\" (UID: \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:38:57.199475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.199442 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k84mx\" (UniqueName: \"kubernetes.io/projected/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-kube-api-access-k84mx\") pod \"maas-api-revoke-sub-1f3ff2b5-c9zdk\" (UID: \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:38:57.338638 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:57.338556 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:38:58.017229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:58.017187 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk"] Oct 30 16:38:58.017684 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:38:58.017653 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd6eb02_0ee3_4854_8b15_d391d7f74b66.slice/crio-f6832ff96948af08c7ee68f02489d59a6150017929c8d9ac8821380dd1020d8f WatchSource:0}: Error finding container f6832ff96948af08c7ee68f02489d59a6150017929c8d9ac8821380dd1020d8f: Status 404 returned error can't find the container with id f6832ff96948af08c7ee68f02489d59a6150017929c8d9ac8821380dd1020d8f Oct 30 16:38:58.138484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:58.138444 5838 generic.go:356] "Generic (PLEG): container finished" podID="cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5" containerID="1caa717c263423c75e99c675b43bc367086791b0b629ca11ff10bfd61f59b555" exitCode=0 Oct 30 16:38:58.138896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:58.138489 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" event={"ID":"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5","Type":"ContainerDied","Data":"1caa717c263423c75e99c675b43bc367086791b0b629ca11ff10bfd61f59b555"} Oct 30 16:38:58.140407 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:58.140375 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" event={"ID":"5bd6eb02-0ee3-4854-8b15-d391d7f74b66","Type":"ContainerStarted","Data":"f6832ff96948af08c7ee68f02489d59a6150017929c8d9ac8821380dd1020d8f"} Oct 30 16:38:58.226382 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:58.226352 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2"] Oct 30 16:38:58.254479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:58.254443 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v"] Oct 30 16:38:58.260042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:58.260013 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt"] Oct 30 16:38:58.265037 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:58.265006 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2e0784-d7z2v"] Oct 30 16:38:58.397925 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:58.397846 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d712e413-514d-4576-ae0a-c80e207ecab7" path="/var/lib/kubelet/pods/d712e413-514d-4576-ae0a-c80e207ecab7/volumes" Oct 30 16:38:58.947994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:58.945120 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" podUID="a1a40be2-e7d1-49f9-a304-2623278732ba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:38:58.947994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:58.945725 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" podUID="0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:38:59.152518 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.152484 5838 generic.go:356] "Generic (PLEG): container finished" podID="5bd6eb02-0ee3-4854-8b15-d391d7f74b66" containerID="dcade81ba514f0025d77737f283f3dae0accf131f2d107fae925b06163eb53dd" exitCode=0 Oct 30 16:38:59.153051 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.152589 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" event={"ID":"5bd6eb02-0ee3-4854-8b15-d391d7f74b66","Type":"ContainerDied","Data":"dcade81ba514f0025d77737f283f3dae0accf131f2d107fae925b06163eb53dd"} Oct 30 16:38:59.314762 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.314730 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:38:59.364397 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.364361 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-kube-api-access-fgnl6\" (UniqueName: \"kubernetes.io/projected/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-kube-api-access-fgnl6\") pod \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\" (UID: \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\") " Oct 30 16:38:59.364741 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.364720 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-openshift-ca\") pod \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\" (UID: \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\") " Oct 30 16:38:59.364951 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.364933 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-xks-ca\") pod \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\" (UID: \"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5\") " Oct 30 16:38:59.368488 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.367343 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-openshift-ca" pod "cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5" (UID: "cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:59.368758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.368724 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-xks-ca" pod "cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5" (UID: "cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:59.405420 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.405375 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-kube-api-access-fgnl6" pod "cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5" (UID: "cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5"). InnerVolumeSpecName "kube-api-access-fgnl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:59.467229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.467145 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgnl6\" (UniqueName: \"kubernetes.io/projected/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-kube-api-access-fgnl6\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:59.467229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.467177 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:59.467229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.467188 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:38:59.969310 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.960955 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp"] Oct 30 16:38:59.969310 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.962032 5838 cpu_manager.go:414] "containerMap: removing container" podUID="cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5" containerName="revoke-keys" Oct 30 16:38:59.969310 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.962053 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5" containerName="revoke-keys" Oct 30 16:38:59.969310 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.962236 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5" containerName="revoke-keys" Oct 30 16:38:59.969310 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:38:59.968514 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:00.013081 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.010837 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp"] Oct 30 16:39:00.100996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.100507 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdr9c\" (UniqueName: \"kubernetes.io/projected/b89f1441-59fb-4d01-a5be-adf923324ff4-kube-api-access-zdr9c\") pod \"maas-api-revoke-sub-d57b0ebf-k7hvp\" (UID: \"b89f1441-59fb-4d01-a5be-adf923324ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:00.100996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.100620 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d57b0ebf-k7hvp\" (UID: \"b89f1441-59fb-4d01-a5be-adf923324ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:00.100996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.100659 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d57b0ebf-k7hvp\" (UID: \"b89f1441-59fb-4d01-a5be-adf923324ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:00.217007 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.204047 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdr9c\" (UniqueName: \"kubernetes.io/projected/b89f1441-59fb-4d01-a5be-adf923324ff4-kube-api-access-zdr9c\") pod \"maas-api-revoke-sub-d57b0ebf-k7hvp\" (UID: \"b89f1441-59fb-4d01-a5be-adf923324ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:00.217007 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.204149 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d57b0ebf-k7hvp\" (UID: \"b89f1441-59fb-4d01-a5be-adf923324ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:00.217007 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.204179 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d57b0ebf-k7hvp\" (UID: \"b89f1441-59fb-4d01-a5be-adf923324ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:00.249052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.236191 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d57b0ebf-k7hvp\" (UID: \"b89f1441-59fb-4d01-a5be-adf923324ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:00.306991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.306609 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdr9c\" (UniqueName: \"kubernetes.io/projected/b89f1441-59fb-4d01-a5be-adf923324ff4-kube-api-access-zdr9c\") pod \"maas-api-revoke-sub-d57b0ebf-k7hvp\" (UID: \"b89f1441-59fb-4d01-a5be-adf923324ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:00.376993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.363482 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d57b0ebf-k7hvp\" (UID: \"b89f1441-59fb-4d01-a5be-adf923324ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:00.376993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.375659 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:00.576498 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.576419 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" Oct 30 16:39:00.591865 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.576638 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4" event={"ID":"cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5","Type":"ContainerDied","Data":"a2a21bff88e043f7ba7a88f3ee2e438966d7a193171f1dafa8af3731c66d4f32"} Oct 30 16:39:00.592562 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.592088 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2a21bff88e043f7ba7a88f3ee2e438966d7a193171f1dafa8af3731c66d4f32" Oct 30 16:39:00.829176 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.829075 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57"] Oct 30 16:39:00.991067 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:00.991026 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r"] Oct 30 16:39:01.396608 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:01.367690 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:39:01.396608 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:01.382508 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.639994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:01.639323 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.731204 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:01.731072 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.746647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:01.746603 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:02.174522 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:02.152096 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:02.196077 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:02.176170 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 16:39:02.196077 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:02.176874 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:39:02.602165 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:02.551303 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:02.602165 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:02.573087 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:02.856221 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:02.856084 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:03.199106 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:03.181051 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.218995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:03.204748 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.350004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:03.341017 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.532906 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:03.503685 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 16:39:03.597004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:03.561879 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 16:39:03.617018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:03.613896 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.959011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:03.958243 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.959011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:03.958636 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:03.959011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:03.958676 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.993991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:03.982603 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded" Oct 30 16:39:04.061336 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.015750 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" podUID="ec27bb3d-b7dd-4e18-9afb-1a76720ae40a" containerName="payload-pre-processing" containerID="cri-o://5489effa2ff750e1c47ca8978d4024d02c4aa6add21b6c6801aae975c404c8ba" gracePeriod=25 Oct 30 16:39:04.061336 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.015935 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:39:04.061336 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.022924 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:39:04.109737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.045418 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 16:39:04.109737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.098490 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" podUID="594c758e-bf83-4687-bb4f-32d89ed6ee53" containerName="payload-processing" containerID="cri-o://2743d3299cd1aa288f8bfb861c48eb1ecc60e23e579de2584e09e7f56da4b839" gracePeriod=25 Oct 30 16:39:04.112883 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:04.112854 5838 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.71s" Oct 30 16:39:04.113178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.113158 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:39:04.113339 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.113324 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:39:04.113480 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.113467 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:39:04.113624 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.113610 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:39:04.113835 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.113818 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:39:04.114009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.113958 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:39:04.114166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.114150 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:39:04.176997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.175413 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp"] Oct 30 16:39:04.245058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.220479 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Oct 30 16:39:04.245058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.220866 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:39:04.264149 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.264111 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:39:04.304996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.301014 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:39:04.323506 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.320697 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:39:04.323506 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.322064 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:39:04.323506 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.322315 5838 generic.go:356] "Generic (PLEG): container finished" podID="0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98" containerID="e77f525c1869f3c4318c540dec0abfc3c9492e9b0506692786cdf96c5caeed5e" exitCode=0 Oct 30 16:39:04.323506 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.322725 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" event={"ID":"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98","Type":"ContainerDied","Data":"e77f525c1869f3c4318c540dec0abfc3c9492e9b0506692786cdf96c5caeed5e"} Oct 30 16:39:04.442080 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.442053 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:39:04.446792 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.446769 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:39:04.464211 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.464173 5838 generic.go:356] "Generic (PLEG): container finished" podID="a1a40be2-e7d1-49f9-a304-2623278732ba" containerID="73cccef9c8dece852b7aa5213fd03e58abe3387099d5f53e4cf4867aab0e7d08" exitCode=0 Oct 30 16:39:04.505117 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.502122 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-kube-api-access-k84mx\" (UniqueName: \"kubernetes.io/projected/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-kube-api-access-k84mx\") pod \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\" (UID: \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\") " Oct 30 16:39:04.505117 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.502179 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-xks-ca\") pod \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\" (UID: \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\") " Oct 30 16:39:04.505117 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.502254 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-openshift-ca\") pod \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\" (UID: \"5bd6eb02-0ee3-4854-8b15-d391d7f74b66\") " Oct 30 16:39:04.509379 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.506320 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-kube-api-access-k84mx" pod "5bd6eb02-0ee3-4854-8b15-d391d7f74b66" (UID: "5bd6eb02-0ee3-4854-8b15-d391d7f74b66"). InnerVolumeSpecName "kube-api-access-k84mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:04.509672 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.509642 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-openshift-ca" pod "5bd6eb02-0ee3-4854-8b15-d391d7f74b66" (UID: "5bd6eb02-0ee3-4854-8b15-d391d7f74b66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:04.517016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.516945 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-xks-ca" pod "5bd6eb02-0ee3-4854-8b15-d391d7f74b66" (UID: "5bd6eb02-0ee3-4854-8b15-d391d7f74b66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:04.529153 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.529128 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:39:04.532249 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.532228 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:39:04.532756 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.532650 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" event={"ID":"b89f1441-59fb-4d01-a5be-adf923324ff4","Type":"ContainerStarted","Data":"83719d3b711a5ad77ce37367f493ab62f8969e2c537032faa4cc1398d2978c4a"} Oct 30 16:39:04.532948 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.532928 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" event={"ID":"a1a40be2-e7d1-49f9-a304-2623278732ba","Type":"ContainerDied","Data":"73cccef9c8dece852b7aa5213fd03e58abe3387099d5f53e4cf4867aab0e7d08"} Oct 30 16:39:04.604909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.604828 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k84mx\" (UniqueName: \"kubernetes.io/projected/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-kube-api-access-k84mx\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:04.604909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.604868 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:04.604909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.604884 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd6eb02-0ee3-4854-8b15-d391d7f74b66-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:04.663821 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.663792 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" Oct 30 16:39:04.722627 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.722596 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" Oct 30 16:39:04.807626 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.807590 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-kube-api-access-87gmp\" (UniqueName: \"kubernetes.io/projected/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-kube-api-access-87gmp\") pod \"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98\" (UID: \"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98\") " Oct 30 16:39:04.807833 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.807819 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a40be2-e7d1-49f9-a304-2623278732ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a40be2-e7d1-49f9-a304-2623278732ba-plugins-config-volume\") pod \"a1a40be2-e7d1-49f9-a304-2623278732ba\" (UID: \"a1a40be2-e7d1-49f9-a304-2623278732ba\") " Oct 30 16:39:04.807908 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.807879 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a40be2-e7d1-49f9-a304-2623278732ba-kube-api-access-pmq6f\" (UniqueName: \"kubernetes.io/projected/a1a40be2-e7d1-49f9-a304-2623278732ba-kube-api-access-pmq6f\") pod \"a1a40be2-e7d1-49f9-a304-2623278732ba\" (UID: \"a1a40be2-e7d1-49f9-a304-2623278732ba\") " Oct 30 16:39:04.807988 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.807934 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-plugins-config-volume\") pod \"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98\" (UID: \"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98\") " Oct 30 16:39:04.808447 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.808412 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a40be2-e7d1-49f9-a304-2623278732ba-plugins-config-volume" pod "a1a40be2-e7d1-49f9-a304-2623278732ba" (UID: "a1a40be2-e7d1-49f9-a304-2623278732ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:04.808597 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.808446 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-plugins-config-volume" pod "0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98" (UID: "0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:04.808658 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.808637 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a40be2-e7d1-49f9-a304-2623278732ba-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:04.808702 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.808659 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:04.814645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.814607 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-kube-api-access-87gmp" pod "0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98" (UID: "0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98"). InnerVolumeSpecName "kube-api-access-87gmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:04.814818 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.814667 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a40be2-e7d1-49f9-a304-2623278732ba-kube-api-access-pmq6f" pod "a1a40be2-e7d1-49f9-a304-2623278732ba" (UID: "a1a40be2-e7d1-49f9-a304-2623278732ba"). InnerVolumeSpecName "kube-api-access-pmq6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:04.912917 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.912775 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmq6f\" (UniqueName: \"kubernetes.io/projected/a1a40be2-e7d1-49f9-a304-2623278732ba-kube-api-access-pmq6f\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:04.912917 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.912816 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87gmp\" (UniqueName: \"kubernetes.io/projected/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98-kube-api-access-87gmp\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:04.988653 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.988356 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" podUID="594c758e-bf83-4687-bb4f-32d89ed6ee53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:39:04.990592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:04.990546 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" podUID="ec27bb3d-b7dd-4e18-9afb-1a76720ae40a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:39:05.490226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.490194 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" Oct 30 16:39:05.490226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.490209 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m" event={"ID":"0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98","Type":"ContainerDied","Data":"bc3bdc49116049c937629187a951392b3c45e4b08dd6e05822c1369617c87f3e"} Oct 30 16:39:05.490450 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.490311 5838 scope.go:122] "RemoveContainer" containerID="e77f525c1869f3c4318c540dec0abfc3c9492e9b0506692786cdf96c5caeed5e" Oct 30 16:39:05.495939 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.495920 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" Oct 30 16:39:05.496100 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.495955 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk" event={"ID":"5bd6eb02-0ee3-4854-8b15-d391d7f74b66","Type":"ContainerDied","Data":"f6832ff96948af08c7ee68f02489d59a6150017929c8d9ac8821380dd1020d8f"} Oct 30 16:39:05.496100 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.496045 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6832ff96948af08c7ee68f02489d59a6150017929c8d9ac8821380dd1020d8f" Oct 30 16:39:05.498169 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.498145 5838 generic.go:356] "Generic (PLEG): container finished" podID="b89f1441-59fb-4d01-a5be-adf923324ff4" containerID="8e6791fb634445b2a39cdf30841360e846c44a7542e1db2985024af297ef9a65" exitCode=0 Oct 30 16:39:05.498267 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.498217 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" event={"ID":"b89f1441-59fb-4d01-a5be-adf923324ff4","Type":"ContainerDied","Data":"8e6791fb634445b2a39cdf30841360e846c44a7542e1db2985024af297ef9a65"} Oct 30 16:39:05.499885 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.499864 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" Oct 30 16:39:05.500042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.499907 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p" event={"ID":"a1a40be2-e7d1-49f9-a304-2623278732ba","Type":"ContainerDied","Data":"d083c6cf5e8aed37d69948015a816a82613cbfff4556bd921af30df0913289ae"} Oct 30 16:39:05.502751 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.502732 5838 scope.go:122] "RemoveContainer" containerID="73cccef9c8dece852b7aa5213fd03e58abe3387099d5f53e4cf4867aab0e7d08" Oct 30 16:39:05.560277 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.560244 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m"] Oct 30 16:39:05.571113 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.571074 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2e0784-667db46bd7-tjx6m"] Oct 30 16:39:05.657927 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.657849 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p"] Oct 30 16:39:05.663115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:05.663089 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2e0784-8ccfd88cc-qv97p"] Oct 30 16:39:06.310994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.298176 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89"] Oct 30 16:39:06.310994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.299272 5838 cpu_manager.go:414] "containerMap: removing container" podUID="0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98" containerName="payload-pre-processing" Oct 30 16:39:06.310994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.299295 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98" containerName="payload-pre-processing" Oct 30 16:39:06.310994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.299324 5838 cpu_manager.go:414] "containerMap: removing container" podUID="a1a40be2-e7d1-49f9-a304-2623278732ba" containerName="payload-processing" Oct 30 16:39:06.310994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.299334 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a40be2-e7d1-49f9-a304-2623278732ba" containerName="payload-processing" Oct 30 16:39:06.310994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.299392 5838 cpu_manager.go:414] "containerMap: removing container" podUID="5bd6eb02-0ee3-4854-8b15-d391d7f74b66" containerName="revoke-keys" Oct 30 16:39:06.310994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.299399 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd6eb02-0ee3-4854-8b15-d391d7f74b66" containerName="revoke-keys" Oct 30 16:39:06.310994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.299554 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd6eb02-0ee3-4854-8b15-d391d7f74b66" containerName="revoke-keys" Oct 30 16:39:06.310994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.299577 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98" containerName="payload-pre-processing" Oct 30 16:39:06.310994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.299587 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a40be2-e7d1-49f9-a304-2623278732ba" containerName="payload-processing" Oct 30 16:39:06.310994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.305448 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:06.417900 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.417835 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98" path="/var/lib/kubelet/pods/0f49c751-3b87-4bd8-bc8f-cdc4e2dc2a98/volumes" Oct 30 16:39:06.418588 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.418566 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a40be2-e7d1-49f9-a304-2623278732ba" path="/var/lib/kubelet/pods/a1a40be2-e7d1-49f9-a304-2623278732ba/volumes" Oct 30 16:39:06.419126 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.419098 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89"] Oct 30 16:39:06.438254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.438217 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08677558-vmp89\" (UID: \"ac57fd23-359e-4db4-b110-2325b4e4960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:06.438451 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.438289 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08677558-vmp89\" (UID: \"ac57fd23-359e-4db4-b110-2325b4e4960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:06.438552 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.438535 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gz6x\" (UniqueName: \"kubernetes.io/projected/ac57fd23-359e-4db4-b110-2325b4e4960e-kube-api-access-7gz6x\") pod \"maas-api-revoke-sub-08677558-vmp89\" (UID: \"ac57fd23-359e-4db4-b110-2325b4e4960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:06.540069 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.539878 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08677558-vmp89\" (UID: \"ac57fd23-359e-4db4-b110-2325b4e4960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:06.540069 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.539998 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08677558-vmp89\" (UID: \"ac57fd23-359e-4db4-b110-2325b4e4960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:06.540292 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.540192 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gz6x\" (UniqueName: \"kubernetes.io/projected/ac57fd23-359e-4db4-b110-2325b4e4960e-kube-api-access-7gz6x\") pod \"maas-api-revoke-sub-08677558-vmp89\" (UID: \"ac57fd23-359e-4db4-b110-2325b4e4960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:06.541180 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.541154 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08677558-vmp89\" (UID: \"ac57fd23-359e-4db4-b110-2325b4e4960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:06.542817 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.542790 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08677558-vmp89\" (UID: \"ac57fd23-359e-4db4-b110-2325b4e4960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:06.570349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.570319 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gz6x\" (UniqueName: \"kubernetes.io/projected/ac57fd23-359e-4db4-b110-2325b4e4960e-kube-api-access-7gz6x\") pod \"maas-api-revoke-sub-08677558-vmp89\" (UID: \"ac57fd23-359e-4db4-b110-2325b4e4960e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:06.659315 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.659288 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:06.772043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.758723 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-openshift-ca\") pod \"b89f1441-59fb-4d01-a5be-adf923324ff4\" (UID: \"b89f1441-59fb-4d01-a5be-adf923324ff4\") " Oct 30 16:39:06.772043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.758800 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b89f1441-59fb-4d01-a5be-adf923324ff4-kube-api-access-zdr9c\" (UniqueName: \"kubernetes.io/projected/b89f1441-59fb-4d01-a5be-adf923324ff4-kube-api-access-zdr9c\") pod \"b89f1441-59fb-4d01-a5be-adf923324ff4\" (UID: \"b89f1441-59fb-4d01-a5be-adf923324ff4\") " Oct 30 16:39:06.772043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.759029 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-xks-ca\") pod \"b89f1441-59fb-4d01-a5be-adf923324ff4\" (UID: \"b89f1441-59fb-4d01-a5be-adf923324ff4\") " Oct 30 16:39:06.815031 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.789440 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:06.815031 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.789875 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-openshift-ca" pod "b89f1441-59fb-4d01-a5be-adf923324ff4" (UID: "b89f1441-59fb-4d01-a5be-adf923324ff4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:06.880265 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.870698 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:06.913671 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.913621 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89f1441-59fb-4d01-a5be-adf923324ff4-kube-api-access-zdr9c" pod "b89f1441-59fb-4d01-a5be-adf923324ff4" (UID: "b89f1441-59fb-4d01-a5be-adf923324ff4"). InnerVolumeSpecName "kube-api-access-zdr9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:06.949550 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.914102 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-xks-ca" pod "b89f1441-59fb-4d01-a5be-adf923324ff4" (UID: "b89f1441-59fb-4d01-a5be-adf923324ff4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:06.974161 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.974121 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdr9c\" (UniqueName: \"kubernetes.io/projected/b89f1441-59fb-4d01-a5be-adf923324ff4-kube-api-access-zdr9c\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:06.974161 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:06.974162 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89f1441-59fb-4d01-a5be-adf923324ff4-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:07.280728 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:07.280699 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89"] Oct 30 16:39:07.281439 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:39:07.281412 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac57fd23_359e_4db4_b110_2325b4e4960e.slice/crio-cf5af8efdce3789dcd13c4f8f6b2bfd01c2a0326748a622a93fda411c4c391e2 WatchSource:0}: Error finding container cf5af8efdce3789dcd13c4f8f6b2bfd01c2a0326748a622a93fda411c4c391e2: Status 404 returned error can't find the container with id cf5af8efdce3789dcd13c4f8f6b2bfd01c2a0326748a622a93fda411c4c391e2 Oct 30 16:39:07.521603 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:07.521571 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" event={"ID":"ac57fd23-359e-4db4-b110-2325b4e4960e","Type":"ContainerStarted","Data":"cf5af8efdce3789dcd13c4f8f6b2bfd01c2a0326748a622a93fda411c4c391e2"} Oct 30 16:39:07.523057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:07.523035 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" Oct 30 16:39:07.523057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:07.523052 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp" event={"ID":"b89f1441-59fb-4d01-a5be-adf923324ff4","Type":"ContainerDied","Data":"83719d3b711a5ad77ce37367f493ab62f8969e2c537032faa4cc1398d2978c4a"} Oct 30 16:39:07.523229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:07.523093 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83719d3b711a5ad77ce37367f493ab62f8969e2c537032faa4cc1398d2978c4a" Oct 30 16:39:08.348139 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:08.348085 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv"] Oct 30 16:39:08.348449 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:08.348348 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerName="istio-proxy" containerID="cri-o://a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0" gracePeriod=30 Oct 30 16:39:08.357895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:08.357861 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw"] Oct 30 16:39:08.358260 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:08.358211 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerName="istio-proxy" containerID="cri-o://f1d54b292dc3ce76921d02b6ceb56166d19df6edf83a0de6d0483ddc40e0845c" gracePeriod=30 Oct 30 16:39:08.364998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:08.364952 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v"] Oct 30 16:39:08.365201 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:08.365180 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" podUID="5a2dbaee-1611-4433-b37c-90487d456baf" containerName="istio-proxy" containerID="cri-o://d5bf51754a7e8e256adfbc38a78352ac17e58ec1818db70bc955d12c3cf1c121" gracePeriod=30 Oct 30 16:39:08.370409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:08.370381 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv"] Oct 30 16:39:08.370631 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:08.370608 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" podUID="318405fd-d020-4bb2-b582-44c2517639ef" containerName="istio-proxy" containerID="cri-o://eb6848775914395bc16c85d97e73a0a2731c4ae1dc687b057e39a048e6332122" gracePeriod=30 Oct 30 16:39:08.605015 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:08.604935 5838 generic.go:356] "Generic (PLEG): container finished" podID="ac57fd23-359e-4db4-b110-2325b4e4960e" containerID="89c9209ee2b8d14d0aa8e20f2325eba570a308ed8b1602829c80f2d2a1b361d7" exitCode=0 Oct 30 16:39:08.605506 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:08.605041 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" event={"ID":"ac57fd23-359e-4db4-b110-2325b4e4960e","Type":"ContainerDied","Data":"89c9209ee2b8d14d0aa8e20f2325eba570a308ed8b1602829c80f2d2a1b361d7"} Oct 30 16:39:08.980087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:08.978162 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd"] Oct 30 16:39:09.024494 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.022743 5838 cpu_manager.go:414] "containerMap: removing container" podUID="b89f1441-59fb-4d01-a5be-adf923324ff4" containerName="revoke-keys" Oct 30 16:39:09.024494 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.022772 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b89f1441-59fb-4d01-a5be-adf923324ff4" containerName="revoke-keys" Oct 30 16:39:09.024494 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.023146 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="b89f1441-59fb-4d01-a5be-adf923324ff4" containerName="revoke-keys" Oct 30 16:39:09.040316 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.034212 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd"] Oct 30 16:39:09.040316 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.034351 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:09.138040 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.138004 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l274\" (UniqueName: \"kubernetes.io/projected/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-kube-api-access-2l274\") pod \"maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd\" (UID: \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:09.138265 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.138241 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd\" (UID: \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:09.138401 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.138382 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd\" (UID: \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:09.239252 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.239218 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd\" (UID: \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:09.239441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.239273 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l274\" (UniqueName: \"kubernetes.io/projected/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-kube-api-access-2l274\") pod \"maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd\" (UID: \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:09.239441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.239424 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd\" (UID: \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:09.240056 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.240031 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd\" (UID: \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:09.241704 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.241674 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd\" (UID: \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:09.267138 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.267108 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l274\" (UniqueName: \"kubernetes.io/projected/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-kube-api-access-2l274\") pod \"maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd\" (UID: \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:09.398003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.397940 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:09.712183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.712155 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd"] Oct 30 16:39:09.717466 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:39:09.717423 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab729303_64bc_42b4_8f35_fa9d5aa5ca66.slice/crio-6eaf1d5cb5f6a613cfd873e19c099884311702e9b13006276d39228823341511 WatchSource:0}: Error finding container 6eaf1d5cb5f6a613cfd873e19c099884311702e9b13006276d39228823341511: Status 404 returned error can't find the container with id 6eaf1d5cb5f6a613cfd873e19c099884311702e9b13006276d39228823341511 Oct 30 16:39:09.766721 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.766697 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:09.848560 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.848522 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac57fd23-359e-4db4-b110-2325b4e4960e-kube-api-access-7gz6x\" (UniqueName: \"kubernetes.io/projected/ac57fd23-359e-4db4-b110-2325b4e4960e-kube-api-access-7gz6x\") pod \"ac57fd23-359e-4db4-b110-2325b4e4960e\" (UID: \"ac57fd23-359e-4db4-b110-2325b4e4960e\") " Oct 30 16:39:09.848729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.848616 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-openshift-ca\") pod \"ac57fd23-359e-4db4-b110-2325b4e4960e\" (UID: \"ac57fd23-359e-4db4-b110-2325b4e4960e\") " Oct 30 16:39:09.848729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.848685 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-xks-ca\") pod \"ac57fd23-359e-4db4-b110-2325b4e4960e\" (UID: \"ac57fd23-359e-4db4-b110-2325b4e4960e\") " Oct 30 16:39:09.849260 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.849230 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-openshift-ca" pod "ac57fd23-359e-4db4-b110-2325b4e4960e" (UID: "ac57fd23-359e-4db4-b110-2325b4e4960e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:09.921042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.918703 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-xks-ca" pod "ac57fd23-359e-4db4-b110-2325b4e4960e" (UID: "ac57fd23-359e-4db4-b110-2325b4e4960e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:09.921042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.919626 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac57fd23-359e-4db4-b110-2325b4e4960e-kube-api-access-7gz6x" pod "ac57fd23-359e-4db4-b110-2325b4e4960e" (UID: "ac57fd23-359e-4db4-b110-2325b4e4960e"). InnerVolumeSpecName "kube-api-access-7gz6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:09.956058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.951237 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gz6x\" (UniqueName: \"kubernetes.io/projected/ac57fd23-359e-4db4-b110-2325b4e4960e-kube-api-access-7gz6x\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:09.956058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.951273 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:09.956058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:09.951288 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac57fd23-359e-4db4-b110-2325b4e4960e-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:10.259062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.259030 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr"] Oct 30 16:39:10.259677 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.259662 5838 cpu_manager.go:414] "containerMap: removing container" podUID="ac57fd23-359e-4db4-b110-2325b4e4960e" containerName="revoke-keys" Oct 30 16:39:10.259737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.259678 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac57fd23-359e-4db4-b110-2325b4e4960e" containerName="revoke-keys" Oct 30 16:39:10.259813 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.259804 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac57fd23-359e-4db4-b110-2325b4e4960e" containerName="revoke-keys" Oct 30 16:39:10.263646 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.263630 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:10.283064 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.283038 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr"] Oct 30 16:39:10.356543 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.356503 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9q7s\" (UniqueName: \"kubernetes.io/projected/136e54ab-b8a1-46a4-8c4e-734d52222000-kube-api-access-q9q7s\") pod \"maas-api-revoke-sub-4a87bebc-ntmxr\" (UID: \"136e54ab-b8a1-46a4-8c4e-734d52222000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:10.356721 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.356698 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a87bebc-ntmxr\" (UID: \"136e54ab-b8a1-46a4-8c4e-734d52222000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:10.356807 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.356792 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a87bebc-ntmxr\" (UID: \"136e54ab-b8a1-46a4-8c4e-734d52222000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:10.458489 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.458452 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a87bebc-ntmxr\" (UID: \"136e54ab-b8a1-46a4-8c4e-734d52222000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:10.458671 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.458537 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a87bebc-ntmxr\" (UID: \"136e54ab-b8a1-46a4-8c4e-734d52222000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:10.458671 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.458656 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9q7s\" (UniqueName: \"kubernetes.io/projected/136e54ab-b8a1-46a4-8c4e-734d52222000-kube-api-access-q9q7s\") pod \"maas-api-revoke-sub-4a87bebc-ntmxr\" (UID: \"136e54ab-b8a1-46a4-8c4e-734d52222000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:10.459231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.459204 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a87bebc-ntmxr\" (UID: \"136e54ab-b8a1-46a4-8c4e-734d52222000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:10.461127 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.461108 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a87bebc-ntmxr\" (UID: \"136e54ab-b8a1-46a4-8c4e-734d52222000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:10.481672 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.481601 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9q7s\" (UniqueName: \"kubernetes.io/projected/136e54ab-b8a1-46a4-8c4e-734d52222000-kube-api-access-q9q7s\") pod \"maas-api-revoke-sub-4a87bebc-ntmxr\" (UID: \"136e54ab-b8a1-46a4-8c4e-734d52222000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:10.588903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.588870 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:10.627507 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.627479 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" Oct 30 16:39:10.627507 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.627490 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89" event={"ID":"ac57fd23-359e-4db4-b110-2325b4e4960e","Type":"ContainerDied","Data":"cf5af8efdce3789dcd13c4f8f6b2bfd01c2a0326748a622a93fda411c4c391e2"} Oct 30 16:39:10.627786 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.627556 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf5af8efdce3789dcd13c4f8f6b2bfd01c2a0326748a622a93fda411c4c391e2" Oct 30 16:39:10.629565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.629532 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" event={"ID":"ab729303-64bc-42b4-8f35-fa9d5aa5ca66","Type":"ContainerStarted","Data":"91f3c80f562d51e77d019b0c138be134d7dd2cec37f7105887c712b4de379fe0"} Oct 30 16:39:10.629736 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.629585 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" event={"ID":"ab729303-64bc-42b4-8f35-fa9d5aa5ca66","Type":"ContainerStarted","Data":"6eaf1d5cb5f6a613cfd873e19c099884311702e9b13006276d39228823341511"} Oct 30 16:39:10.671145 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.671094 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" podStartSLOduration=2.671077941 podStartE2EDuration="2.671077941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:10.667601621 +0000 UTC m=+1112.416191037" watchObservedRunningTime="2026-10-30 16:39:10.671077941 +0000 UTC m=+1112.419667327" Oct 30 16:39:10.901618 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:10.901586 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr"] Oct 30 16:39:10.902552 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:39:10.902526 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136e54ab_b8a1_46a4_8c4e_734d52222000.slice/crio-cb98edc3f1d649a4363e3d1c709e2d9e8002612ee140a91c72c25e7b70009e5b WatchSource:0}: Error finding container cb98edc3f1d649a4363e3d1c709e2d9e8002612ee140a91c72c25e7b70009e5b: Status 404 returned error can't find the container with id cb98edc3f1d649a4363e3d1c709e2d9e8002612ee140a91c72c25e7b70009e5b Oct 30 16:39:11.336178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:11.336144 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:39:11.336329 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:11.336198 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" podUID="318405fd-d020-4bb2-b582-44c2517639ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:39:11.640320 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:11.640277 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" event={"ID":"136e54ab-b8a1-46a4-8c4e-734d52222000","Type":"ContainerStarted","Data":"3eaca676701827609d03dfda05e808b2358a91e7a95645c8f28d18f47baba94b"} Oct 30 16:39:11.640497 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:11.640367 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" event={"ID":"136e54ab-b8a1-46a4-8c4e-734d52222000","Type":"ContainerStarted","Data":"cb98edc3f1d649a4363e3d1c709e2d9e8002612ee140a91c72c25e7b70009e5b"} Oct 30 16:39:11.642039 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:11.642011 5838 generic.go:356] "Generic (PLEG): container finished" podID="ab729303-64bc-42b4-8f35-fa9d5aa5ca66" containerID="91f3c80f562d51e77d019b0c138be134d7dd2cec37f7105887c712b4de379fe0" exitCode=0 Oct 30 16:39:11.642193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:11.642093 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" event={"ID":"ab729303-64bc-42b4-8f35-fa9d5aa5ca66","Type":"ContainerDied","Data":"91f3c80f562d51e77d019b0c138be134d7dd2cec37f7105887c712b4de379fe0"} Oct 30 16:39:11.683824 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:11.683766 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" podStartSLOduration=1.683752004 podStartE2EDuration="1.683752004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:11.679960223 +0000 UTC m=+1113.428549624" watchObservedRunningTime="2026-10-30 16:39:11.683752004 +0000 UTC m=+1113.432341390" Oct 30 16:39:12.321140 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:12.321105 5838 patch_prober.go:29] interesting pod/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:39:12.321583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:12.321167 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" podUID="5a2dbaee-1611-4433-b37c-90487d456baf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:39:12.653357 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:12.653270 5838 generic.go:356] "Generic (PLEG): container finished" podID="136e54ab-b8a1-46a4-8c4e-734d52222000" containerID="3eaca676701827609d03dfda05e808b2358a91e7a95645c8f28d18f47baba94b" exitCode=0 Oct 30 16:39:12.653506 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:12.653353 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" event={"ID":"136e54ab-b8a1-46a4-8c4e-734d52222000","Type":"ContainerDied","Data":"3eaca676701827609d03dfda05e808b2358a91e7a95645c8f28d18f47baba94b"} Oct 30 16:39:12.837298 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:12.837271 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:12.911556 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:12.911476 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-xks-ca\") pod \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\" (UID: \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\") " Oct 30 16:39:12.911780 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:12.911759 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-kube-api-access-2l274\" (UniqueName: \"kubernetes.io/projected/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-kube-api-access-2l274\") pod \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\" (UID: \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\") " Oct 30 16:39:12.911829 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:12.911804 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-openshift-ca\") pod \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\" (UID: \"ab729303-64bc-42b4-8f35-fa9d5aa5ca66\") " Oct 30 16:39:12.912225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:12.912194 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-openshift-ca" pod "ab729303-64bc-42b4-8f35-fa9d5aa5ca66" (UID: "ab729303-64bc-42b4-8f35-fa9d5aa5ca66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:12.912462 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:12.912435 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:12.913781 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:12.913756 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-xks-ca" pod "ab729303-64bc-42b4-8f35-fa9d5aa5ca66" (UID: "ab729303-64bc-42b4-8f35-fa9d5aa5ca66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:12.914034 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:12.914014 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-kube-api-access-2l274" pod "ab729303-64bc-42b4-8f35-fa9d5aa5ca66" (UID: "ab729303-64bc-42b4-8f35-fa9d5aa5ca66"). InnerVolumeSpecName "kube-api-access-2l274". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:13.013577 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.013544 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l274\" (UniqueName: \"kubernetes.io/projected/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-kube-api-access-2l274\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.013577 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.013575 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab729303-64bc-42b4-8f35-fa9d5aa5ca66-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.645459 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.645426 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:39:13.672750 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.672717 5838 generic.go:356] "Generic (PLEG): container finished" podID="5a2dbaee-1611-4433-b37c-90487d456baf" containerID="d5bf51754a7e8e256adfbc38a78352ac17e58ec1818db70bc955d12c3cf1c121" exitCode=0 Oct 30 16:39:13.673036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.672884 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" event={"ID":"5a2dbaee-1611-4433-b37c-90487d456baf","Type":"ContainerDied","Data":"d5bf51754a7e8e256adfbc38a78352ac17e58ec1818db70bc955d12c3cf1c121"} Oct 30 16:39:13.675934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.675907 5838 generic.go:356] "Generic (PLEG): container finished" podID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerID="a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0" exitCode=0 Oct 30 16:39:13.676139 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.676017 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" event={"ID":"5a3985ca-cecb-4c1a-9584-af2320d1c71e","Type":"ContainerDied","Data":"a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0"} Oct 30 16:39:13.676139 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.676037 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" Oct 30 16:39:13.676139 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.676063 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv" event={"ID":"5a3985ca-cecb-4c1a-9584-af2320d1c71e","Type":"ContainerDied","Data":"143c3f2fd3e8452929a8bfeeb02f10bccbb76f457c43f667dfca28d7b77545d9"} Oct 30 16:39:13.676139 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.676091 5838 scope.go:122] "RemoveContainer" containerID="a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0" Oct 30 16:39:13.678496 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.678472 5838 generic.go:356] "Generic (PLEG): container finished" podID="318405fd-d020-4bb2-b582-44c2517639ef" containerID="eb6848775914395bc16c85d97e73a0a2731c4ae1dc687b057e39a048e6332122" exitCode=0 Oct 30 16:39:13.678589 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.678546 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" event={"ID":"318405fd-d020-4bb2-b582-44c2517639ef","Type":"ContainerDied","Data":"eb6848775914395bc16c85d97e73a0a2731c4ae1dc687b057e39a048e6332122"} Oct 30 16:39:13.681681 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.681651 5838 generic.go:356] "Generic (PLEG): container finished" podID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerID="f1d54b292dc3ce76921d02b6ceb56166d19df6edf83a0de6d0483ddc40e0845c" exitCode=0 Oct 30 16:39:13.681804 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.681784 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" event={"ID":"f3ec9020-bd36-43a5-8b7a-8f753f7230b5","Type":"ContainerDied","Data":"f1d54b292dc3ce76921d02b6ceb56166d19df6edf83a0de6d0483ddc40e0845c"} Oct 30 16:39:13.683654 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.683627 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" event={"ID":"ab729303-64bc-42b4-8f35-fa9d5aa5ca66","Type":"ContainerDied","Data":"6eaf1d5cb5f6a613cfd873e19c099884311702e9b13006276d39228823341511"} Oct 30 16:39:13.683774 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.683678 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eaf1d5cb5f6a613cfd873e19c099884311702e9b13006276d39228823341511" Oct 30 16:39:13.683837 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.683803 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd" Oct 30 16:39:13.699266 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.699244 5838 scope.go:122] "RemoveContainer" containerID="a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0" Oct 30 16:39:13.699892 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:13.699802 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0\": container with ID starting with a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0 not found: ID does not exist" containerID="a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0" Oct 30 16:39:13.699892 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.699840 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0"} err="failed to get container status \"a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0\": rpc error: code = NotFound desc = could not find container \"a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0\": container with ID starting with a0ac860ec09f5585961bdb4119fc2e069d55635fd036e62a0343651901208df0 not found: ID does not exist" Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.726899 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-podinfo\") pod \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.727140 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-credential-socket\") pod \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.727190 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istiod-ca-cert\") pod \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.727238 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-kube-api-access-clzqg\" (UniqueName: \"kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-kube-api-access-clzqg\") pod \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.727301 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-envoy\") pod \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.727387 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-certs\") pod \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.727428 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-data\") pod \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.727533 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-token\" (UniqueName: \"kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-token\") pod \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.727609 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-socket\") pod \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\" (UID: \"5a3985ca-cecb-4c1a-9584-af2320d1c71e\") " Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.729420 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-socket" pod "5a3985ca-cecb-4c1a-9584-af2320d1c71e" (UID: "5a3985ca-cecb-4c1a-9584-af2320d1c71e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.732268 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-certs" pod "5a3985ca-cecb-4c1a-9584-af2320d1c71e" (UID: "5a3985ca-cecb-4c1a-9584-af2320d1c71e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.732365 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-credential-socket" pod "5a3985ca-cecb-4c1a-9584-af2320d1c71e" (UID: "5a3985ca-cecb-4c1a-9584-af2320d1c71e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.735011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.732906 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-data" pod "5a3985ca-cecb-4c1a-9584-af2320d1c71e" (UID: "5a3985ca-cecb-4c1a-9584-af2320d1c71e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.763019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.738129 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istiod-ca-cert" pod "5a3985ca-cecb-4c1a-9584-af2320d1c71e" (UID: "5a3985ca-cecb-4c1a-9584-af2320d1c71e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:13.763019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.744083 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-envoy" pod "5a3985ca-cecb-4c1a-9584-af2320d1c71e" (UID: "5a3985ca-cecb-4c1a-9584-af2320d1c71e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.763019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.744874 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-kube-api-access-clzqg" pod "5a3985ca-cecb-4c1a-9584-af2320d1c71e" (UID: "5a3985ca-cecb-4c1a-9584-af2320d1c71e"). InnerVolumeSpecName "kube-api-access-clzqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:13.766129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.766041 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-podinfo" pod "5a3985ca-cecb-4c1a-9584-af2320d1c71e" (UID: "5a3985ca-cecb-4c1a-9584-af2320d1c71e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:13.768234 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.768201 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-token" pod "5a3985ca-cecb-4c1a-9584-af2320d1c71e" (UID: "5a3985ca-cecb-4c1a-9584-af2320d1c71e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:13.792752 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.792292 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s"] Oct 30 16:39:13.798128 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.796808 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:39:13.799874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.799851 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:39:13.803232 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.803211 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:39:13.808515 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.808488 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr"] Oct 30 16:39:13.819161 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.819116 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m"] Oct 30 16:39:13.830951 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.830848 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.830951 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.830879 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.830951 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.830896 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.830951 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.830911 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.830951 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.830932 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.830951 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.830945 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clzqg\" (UniqueName: \"kubernetes.io/projected/5a3985ca-cecb-4c1a-9584-af2320d1c71e-kube-api-access-clzqg\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.831394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.830959 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.831394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.830993 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.831394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.831006 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3985ca-cecb-4c1a-9584-af2320d1c71e-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.931601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.931522 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-socket\") pod \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " Oct 30 16:39:13.931601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.931598 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-envoy\") pod \"5a2dbaee-1611-4433-b37c-90487d456baf\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " Oct 30 16:39:13.931934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.931628 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-podinfo\") pod \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " Oct 30 16:39:13.931934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.931672 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-istio-token\" (UniqueName: \"kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-istio-token\") pod \"5a2dbaee-1611-4433-b37c-90487d456baf\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " Oct 30 16:39:13.931934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.931712 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-credential-socket\") pod \"318405fd-d020-4bb2-b582-44c2517639ef\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " Oct 30 16:39:13.931934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.931745 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istiod-ca-cert\") pod \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " Oct 30 16:39:13.931934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.931782 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-certs\") pod \"5a2dbaee-1611-4433-b37c-90487d456baf\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " Oct 30 16:39:13.931934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.931809 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-envoy\") pod \"318405fd-d020-4bb2-b582-44c2517639ef\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " Oct 30 16:39:13.931934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.931863 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-credential-socket\") pod \"5a2dbaee-1611-4433-b37c-90487d456baf\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " Oct 30 16:39:13.931934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.931896 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-credential-socket\") pod \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " Oct 30 16:39:13.932331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.931942 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-envoy\") pod \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " Oct 30 16:39:13.932331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932047 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a2dbaee-1611-4433-b37c-90487d456baf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a2dbaee-1611-4433-b37c-90487d456baf-istiod-ca-cert\") pod \"5a2dbaee-1611-4433-b37c-90487d456baf\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " Oct 30 16:39:13.932331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932081 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-certs\") pod \"318405fd-d020-4bb2-b582-44c2517639ef\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " Oct 30 16:39:13.932331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932120 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-certs\") pod \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " Oct 30 16:39:13.932331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.931854 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-socket" pod "f3ec9020-bd36-43a5-8b7a-8f753f7230b5" (UID: "f3ec9020-bd36-43a5-8b7a-8f753f7230b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.932331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932093 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-credential-socket" pod "318405fd-d020-4bb2-b582-44c2517639ef" (UID: "318405fd-d020-4bb2-b582-44c2517639ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.932331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932151 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-token\" (UniqueName: \"kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-token\") pod \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " Oct 30 16:39:13.932331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932210 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318405fd-d020-4bb2-b582-44c2517639ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318405fd-d020-4bb2-b582-44c2517639ef-istiod-ca-cert\") pod \"318405fd-d020-4bb2-b582-44c2517639ef\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " Oct 30 16:39:13.932331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932263 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-istio-token\" (UniqueName: \"kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-istio-token\") pod \"318405fd-d020-4bb2-b582-44c2517639ef\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " Oct 30 16:39:13.932331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932323 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-kube-api-access-rkq2s\" (UniqueName: \"kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-kube-api-access-rkq2s\") pod \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " Oct 30 16:39:13.932802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932393 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-socket\") pod \"5a2dbaee-1611-4433-b37c-90487d456baf\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " Oct 30 16:39:13.932802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932458 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-kube-api-access-sz47h\" (UniqueName: \"kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-kube-api-access-sz47h\") pod \"5a2dbaee-1611-4433-b37c-90487d456baf\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " Oct 30 16:39:13.932802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932493 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-kube-api-access-tk2qh\" (UniqueName: \"kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-kube-api-access-tk2qh\") pod \"318405fd-d020-4bb2-b582-44c2517639ef\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " Oct 30 16:39:13.932802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932531 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-data\") pod \"5a2dbaee-1611-4433-b37c-90487d456baf\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " Oct 30 16:39:13.932802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932554 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-data\") pod \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\" (UID: \"f3ec9020-bd36-43a5-8b7a-8f753f7230b5\") " Oct 30 16:39:13.932802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932626 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-socket\") pod \"318405fd-d020-4bb2-b582-44c2517639ef\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " Oct 30 16:39:13.932802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932644 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a2dbaee-1611-4433-b37c-90487d456baf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a2dbaee-1611-4433-b37c-90487d456baf-istio-podinfo\") pod \"5a2dbaee-1611-4433-b37c-90487d456baf\" (UID: \"5a2dbaee-1611-4433-b37c-90487d456baf\") " Oct 30 16:39:13.932802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932664 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/318405fd-d020-4bb2-b582-44c2517639ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318405fd-d020-4bb2-b582-44c2517639ef-istio-podinfo\") pod \"318405fd-d020-4bb2-b582-44c2517639ef\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " Oct 30 16:39:13.932802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.932693 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-data\") pod \"318405fd-d020-4bb2-b582-44c2517639ef\" (UID: \"318405fd-d020-4bb2-b582-44c2517639ef\") " Oct 30 16:39:13.933231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.933198 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.933231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.933215 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:13.933409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.933386 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-data" pod "318405fd-d020-4bb2-b582-44c2517639ef" (UID: "318405fd-d020-4bb2-b582-44c2517639ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.933628 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.933605 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istiod-ca-cert" pod "f3ec9020-bd36-43a5-8b7a-8f753f7230b5" (UID: "f3ec9020-bd36-43a5-8b7a-8f753f7230b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:13.933794 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.933780 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-certs" pod "5a2dbaee-1611-4433-b37c-90487d456baf" (UID: "5a2dbaee-1611-4433-b37c-90487d456baf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.934771 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.934741 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-credential-socket" pod "f3ec9020-bd36-43a5-8b7a-8f753f7230b5" (UID: "f3ec9020-bd36-43a5-8b7a-8f753f7230b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.935808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.935190 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-data" pod "f3ec9020-bd36-43a5-8b7a-8f753f7230b5" (UID: "f3ec9020-bd36-43a5-8b7a-8f753f7230b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.937961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.937651 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-istio-token" pod "5a2dbaee-1611-4433-b37c-90487d456baf" (UID: "5a2dbaee-1611-4433-b37c-90487d456baf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:13.937961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.937752 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-envoy" pod "f3ec9020-bd36-43a5-8b7a-8f753f7230b5" (UID: "f3ec9020-bd36-43a5-8b7a-8f753f7230b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.937961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.937785 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a2dbaee-1611-4433-b37c-90487d456baf-istio-podinfo" pod "5a2dbaee-1611-4433-b37c-90487d456baf" (UID: "5a2dbaee-1611-4433-b37c-90487d456baf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:13.937961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.938027 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-socket" pod "318405fd-d020-4bb2-b582-44c2517639ef" (UID: "318405fd-d020-4bb2-b582-44c2517639ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.938320 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.938112 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2dbaee-1611-4433-b37c-90487d456baf-istiod-ca-cert" pod "5a2dbaee-1611-4433-b37c-90487d456baf" (UID: "5a2dbaee-1611-4433-b37c-90487d456baf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:13.938840 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.938813 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-kube-api-access-tk2qh" pod "318405fd-d020-4bb2-b582-44c2517639ef" (UID: "318405fd-d020-4bb2-b582-44c2517639ef"). InnerVolumeSpecName "kube-api-access-tk2qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:13.938936 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.938815 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-data" pod "5a2dbaee-1611-4433-b37c-90487d456baf" (UID: "5a2dbaee-1611-4433-b37c-90487d456baf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.939047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.939020 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-credential-socket" pod "5a2dbaee-1611-4433-b37c-90487d456baf" (UID: "5a2dbaee-1611-4433-b37c-90487d456baf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.939255 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.939233 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-certs" pod "f3ec9020-bd36-43a5-8b7a-8f753f7230b5" (UID: "f3ec9020-bd36-43a5-8b7a-8f753f7230b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.939459 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.939435 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-socket" pod "5a2dbaee-1611-4433-b37c-90487d456baf" (UID: "5a2dbaee-1611-4433-b37c-90487d456baf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.939716 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.939569 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318405fd-d020-4bb2-b582-44c2517639ef-istiod-ca-cert" pod "318405fd-d020-4bb2-b582-44c2517639ef" (UID: "318405fd-d020-4bb2-b582-44c2517639ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:13.939716 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.939685 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-podinfo" pod "f3ec9020-bd36-43a5-8b7a-8f753f7230b5" (UID: "f3ec9020-bd36-43a5-8b7a-8f753f7230b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:13.939853 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.939757 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-envoy" pod "5a2dbaee-1611-4433-b37c-90487d456baf" (UID: "5a2dbaee-1611-4433-b37c-90487d456baf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.940198 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.940020 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-kube-api-access-sz47h" pod "5a2dbaee-1611-4433-b37c-90487d456baf" (UID: "5a2dbaee-1611-4433-b37c-90487d456baf"). InnerVolumeSpecName "kube-api-access-sz47h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:13.940293 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.940200 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-token" pod "f3ec9020-bd36-43a5-8b7a-8f753f7230b5" (UID: "f3ec9020-bd36-43a5-8b7a-8f753f7230b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:13.942086 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.941988 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-certs" pod "318405fd-d020-4bb2-b582-44c2517639ef" (UID: "318405fd-d020-4bb2-b582-44c2517639ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.942181 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.942105 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/318405fd-d020-4bb2-b582-44c2517639ef-istio-podinfo" pod "318405fd-d020-4bb2-b582-44c2517639ef" (UID: "318405fd-d020-4bb2-b582-44c2517639ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:13.942181 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.942107 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-kube-api-access-rkq2s" pod "f3ec9020-bd36-43a5-8b7a-8f753f7230b5" (UID: "f3ec9020-bd36-43a5-8b7a-8f753f7230b5"). InnerVolumeSpecName "kube-api-access-rkq2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:13.942181 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.942129 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-envoy" pod "318405fd-d020-4bb2-b582-44c2517639ef" (UID: "318405fd-d020-4bb2-b582-44c2517639ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:13.942311 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.942183 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-istio-token" pod "318405fd-d020-4bb2-b582-44c2517639ef" (UID: "318405fd-d020-4bb2-b582-44c2517639ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:13.990053 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:13.990023 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:14.034627 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034593 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkq2s\" (UniqueName: \"kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-kube-api-access-rkq2s\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034627 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034625 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034641 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz47h\" (UniqueName: \"kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-kube-api-access-sz47h\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034655 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk2qh\" (UniqueName: \"kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-kube-api-access-tk2qh\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034667 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034678 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034689 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034701 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a2dbaee-1611-4433-b37c-90487d456baf-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034713 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/318405fd-d020-4bb2-b582-44c2517639ef-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034724 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034737 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034748 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034759 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a2dbaee-1611-4433-b37c-90487d456baf-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034770 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034782 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034794 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034806 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2dbaee-1611-4433-b37c-90487d456baf-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034817 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034828 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034843 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a2dbaee-1611-4433-b37c-90487d456baf-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034856 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/318405fd-d020-4bb2-b582-44c2517639ef-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034870 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.034874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034882 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ec9020-bd36-43a5-8b7a-8f753f7230b5-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.035508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034895 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/318405fd-d020-4bb2-b582-44c2517639ef-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.035508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.034907 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/318405fd-d020-4bb2-b582-44c2517639ef-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.126940 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.126753 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv"] Oct 30 16:39:14.132526 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.132496 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-rpclv"] Oct 30 16:39:14.135820 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.135783 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136e54ab-b8a1-46a4-8c4e-734d52222000-kube-api-access-q9q7s\" (UniqueName: \"kubernetes.io/projected/136e54ab-b8a1-46a4-8c4e-734d52222000-kube-api-access-q9q7s\") pod \"136e54ab-b8a1-46a4-8c4e-734d52222000\" (UID: \"136e54ab-b8a1-46a4-8c4e-734d52222000\") " Oct 30 16:39:14.136049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.135918 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-openshift-ca\") pod \"136e54ab-b8a1-46a4-8c4e-734d52222000\" (UID: \"136e54ab-b8a1-46a4-8c4e-734d52222000\") " Oct 30 16:39:14.136129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.136044 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-xks-ca\") pod \"136e54ab-b8a1-46a4-8c4e-734d52222000\" (UID: \"136e54ab-b8a1-46a4-8c4e-734d52222000\") " Oct 30 16:39:14.136542 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.136518 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-openshift-ca" pod "136e54ab-b8a1-46a4-8c4e-734d52222000" (UID: "136e54ab-b8a1-46a4-8c4e-734d52222000"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:14.138572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.138549 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136e54ab-b8a1-46a4-8c4e-734d52222000-kube-api-access-q9q7s" pod "136e54ab-b8a1-46a4-8c4e-734d52222000" (UID: "136e54ab-b8a1-46a4-8c4e-734d52222000"). InnerVolumeSpecName "kube-api-access-q9q7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:14.140510 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.140484 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-xks-ca" pod "136e54ab-b8a1-46a4-8c4e-734d52222000" (UID: "136e54ab-b8a1-46a4-8c4e-734d52222000"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:14.238312 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.238284 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.238467 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.238316 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136e54ab-b8a1-46a4-8c4e-734d52222000-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.238467 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.238328 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9q7s\" (UniqueName: \"kubernetes.io/projected/136e54ab-b8a1-46a4-8c4e-734d52222000-kube-api-access-q9q7s\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:14.396825 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.396783 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" path="/var/lib/kubelet/pods/5a3985ca-cecb-4c1a-9584-af2320d1c71e/volumes" Oct 30 16:39:14.694005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.693906 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" Oct 30 16:39:14.694005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.693943 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr" event={"ID":"136e54ab-b8a1-46a4-8c4e-734d52222000","Type":"ContainerDied","Data":"cb98edc3f1d649a4363e3d1c709e2d9e8002612ee140a91c72c25e7b70009e5b"} Oct 30 16:39:14.694526 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.694011 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb98edc3f1d649a4363e3d1c709e2d9e8002612ee140a91c72c25e7b70009e5b" Oct 30 16:39:14.695583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.695561 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" Oct 30 16:39:14.695693 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.695591 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v" event={"ID":"5a2dbaee-1611-4433-b37c-90487d456baf","Type":"ContainerDied","Data":"190119d76db8d4a7862bb407562704cede792f02b9b3ad1411d5dbbd6a72e7c9"} Oct 30 16:39:14.695693 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.695656 5838 scope.go:122] "RemoveContainer" containerID="d5bf51754a7e8e256adfbc38a78352ac17e58ec1818db70bc955d12c3cf1c121" Oct 30 16:39:14.698739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.698718 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" Oct 30 16:39:14.698853 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.698722 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv" event={"ID":"318405fd-d020-4bb2-b582-44c2517639ef","Type":"ContainerDied","Data":"cfe09b8fd67631f114640d56c37292ee99102b73a59e249097f8517784f9f78b"} Oct 30 16:39:14.700268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.700247 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" event={"ID":"f3ec9020-bd36-43a5-8b7a-8f753f7230b5","Type":"ContainerDied","Data":"f218dbb22c95c2c570d92ec2b7d2c61a893e48834df732f6229156222e7b490e"} Oct 30 16:39:14.700374 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.700273 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw" Oct 30 16:39:14.711032 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.710884 5838 scope.go:122] "RemoveContainer" containerID="eb6848775914395bc16c85d97e73a0a2731c4ae1dc687b057e39a048e6332122" Oct 30 16:39:14.722095 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.722072 5838 scope.go:122] "RemoveContainer" containerID="f1d54b292dc3ce76921d02b6ceb56166d19df6edf83a0de6d0483ddc40e0845c" Oct 30 16:39:14.751625 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.751592 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw"] Oct 30 16:39:14.757399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.757375 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-nwnjw"] Oct 30 16:39:14.790320 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.790260 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v"] Oct 30 16:39:14.795701 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.795666 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-vdb9v"] Oct 30 16:39:14.828817 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.828774 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv"] Oct 30 16:39:14.833873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.833846 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2e0784-openshift-default-56f9f79fff-fb7mv"] Oct 30 16:39:14.965802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.965707 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" podUID="594c758e-bf83-4687-bb4f-32d89ed6ee53" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:39:14.965802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:14.965724 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" podUID="ec27bb3d-b7dd-4e18-9afb-1a76720ae40a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:39:16.398903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:16.398866 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318405fd-d020-4bb2-b582-44c2517639ef" path="/var/lib/kubelet/pods/318405fd-d020-4bb2-b582-44c2517639ef/volumes" Oct 30 16:39:16.399455 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:16.399438 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a2dbaee-1611-4433-b37c-90487d456baf" path="/var/lib/kubelet/pods/5a2dbaee-1611-4433-b37c-90487d456baf/volumes" Oct 30 16:39:16.399887 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:16.399873 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" path="/var/lib/kubelet/pods/f3ec9020-bd36-43a5-8b7a-8f753f7230b5/volumes" Oct 30 16:39:17.429372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.429332 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" containerID="cri-o://bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea" gracePeriod=44 Oct 30 16:39:17.716841 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.716814 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:39:17.750919 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.750886 5838 generic.go:356] "Generic (PLEG): container finished" podID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerID="bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea" exitCode=0 Oct 30 16:39:17.751098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.750937 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" Oct 30 16:39:17.751098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.750983 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" event={"ID":"f680ef8d-a3ae-493c-b45f-1de23cd0b63a","Type":"ContainerDied","Data":"bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea"} Oct 30 16:39:17.751098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.751070 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r" event={"ID":"f680ef8d-a3ae-493c-b45f-1de23cd0b63a","Type":"ContainerDied","Data":"f9603903ade4bb3ccd857cb665aa0a5b5bcd2809f4b0c7105a7606ef4ee8d9d7"} Oct 30 16:39:17.751098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.751096 5838 scope.go:122] "RemoveContainer" containerID="bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea" Oct 30 16:39:17.763707 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.763671 5838 scope.go:122] "RemoveContainer" containerID="bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea" Oct 30 16:39:17.764264 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:17.764230 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea\": container with ID starting with bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea not found: ID does not exist" containerID="bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea" Oct 30 16:39:17.764380 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.764276 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea"} err="failed to get container status \"bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea\": rpc error: code = NotFound desc = could not find container \"bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea\": container with ID starting with bab08bf52bffef32075b747ae706664c415ead2b41b6f67a3b278c41798500ea not found: ID does not exist" Oct 30 16:39:17.788609 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.788569 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-kube-api-access-qkq5p\" (UniqueName: \"kubernetes.io/projected/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-kube-api-access-qkq5p\") pod \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " Oct 30 16:39:17.788804 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.788737 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tmp-dir\") pod \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " Oct 30 16:39:17.788876 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.788829 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-dshm\") pod \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " Oct 30 16:39:17.788876 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.788858 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-model-cache\") pod \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " Oct 30 16:39:17.789036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.788926 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tls-certs\") pod \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " Oct 30 16:39:17.789036 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.789015 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-home\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-home\") pod \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\" (UID: \"f680ef8d-a3ae-493c-b45f-1de23cd0b63a\") " Oct 30 16:39:17.789597 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.789559 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-model-cache" pod "f680ef8d-a3ae-493c-b45f-1de23cd0b63a" (UID: "f680ef8d-a3ae-493c-b45f-1de23cd0b63a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:17.789849 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.789824 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-home" pod "f680ef8d-a3ae-493c-b45f-1de23cd0b63a" (UID: "f680ef8d-a3ae-493c-b45f-1de23cd0b63a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:17.790088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.790068 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tmp-dir" pod "f680ef8d-a3ae-493c-b45f-1de23cd0b63a" (UID: "f680ef8d-a3ae-493c-b45f-1de23cd0b63a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:17.791683 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.791656 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-dshm" pod "f680ef8d-a3ae-493c-b45f-1de23cd0b63a" (UID: "f680ef8d-a3ae-493c-b45f-1de23cd0b63a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:17.792442 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.792417 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-kube-api-access-qkq5p" pod "f680ef8d-a3ae-493c-b45f-1de23cd0b63a" (UID: "f680ef8d-a3ae-493c-b45f-1de23cd0b63a"). InnerVolumeSpecName "kube-api-access-qkq5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:17.792637 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.792612 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tls-certs" pod "f680ef8d-a3ae-493c-b45f-1de23cd0b63a" (UID: "f680ef8d-a3ae-493c-b45f-1de23cd0b63a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:17.890916 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.890879 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkq5p\" (UniqueName: \"kubernetes.io/projected/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-kube-api-access-qkq5p\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:17.890916 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.890911 5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tmp-dir\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:17.890916 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.890923 5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-dshm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:17.891204 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.890932 5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-model-cache\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:17.891204 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.890941 5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-tls-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:17.891204 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:17.890949 5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f680ef8d-a3ae-493c-b45f-1de23cd0b63a-home\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:18.099647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:18.099613 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r"] Oct 30 16:39:18.104533 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:18.104509 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b8ab9f/simulator-w4-b8ab9f-kserve-578c6cbf97-6pm6r"] Oct 30 16:39:18.397696 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:18.397623 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" path="/var/lib/kubelet/pods/f680ef8d-a3ae-493c-b45f-1de23cd0b63a/volumes" Oct 30 16:39:19.363179 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.363143 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr"] Oct 30 16:39:19.395212 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395175 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg"] Oct 30 16:39:19.395851 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395830 5838 cpu_manager.go:414] "containerMap: removing container" podUID="318405fd-d020-4bb2-b582-44c2517639ef" containerName="istio-proxy" Oct 30 16:39:19.395908 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395855 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318405fd-d020-4bb2-b582-44c2517639ef" containerName="istio-proxy" Oct 30 16:39:19.395908 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395871 5838 cpu_manager.go:414] "containerMap: removing container" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerName="istio-proxy" Oct 30 16:39:19.395908 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395878 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerName="istio-proxy" Oct 30 16:39:19.395908 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395896 5838 cpu_manager.go:414] "containerMap: removing container" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" Oct 30 16:39:19.395908 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395904 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" Oct 30 16:39:19.396087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395916 5838 cpu_manager.go:414] "containerMap: removing container" podUID="5a2dbaee-1611-4433-b37c-90487d456baf" containerName="istio-proxy" Oct 30 16:39:19.396087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395923 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a2dbaee-1611-4433-b37c-90487d456baf" containerName="istio-proxy" Oct 30 16:39:19.396087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395936 5838 cpu_manager.go:414] "containerMap: removing container" podUID="ab729303-64bc-42b4-8f35-fa9d5aa5ca66" containerName="revoke-keys" Oct 30 16:39:19.396087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395941 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab729303-64bc-42b4-8f35-fa9d5aa5ca66" containerName="revoke-keys" Oct 30 16:39:19.396087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395963 5838 cpu_manager.go:414] "containerMap: removing container" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerName="istio-proxy" Oct 30 16:39:19.396087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395983 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerName="istio-proxy" Oct 30 16:39:19.396087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.395996 5838 cpu_manager.go:414] "containerMap: removing container" podUID="136e54ab-b8a1-46a4-8c4e-734d52222000" containerName="revoke-keys" Oct 30 16:39:19.396087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.396003 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136e54ab-b8a1-46a4-8c4e-734d52222000" containerName="revoke-keys" Oct 30 16:39:19.396328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.396125 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab729303-64bc-42b4-8f35-fa9d5aa5ca66" containerName="revoke-keys" Oct 30 16:39:19.396328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.396138 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3985ca-cecb-4c1a-9584-af2320d1c71e" containerName="istio-proxy" Oct 30 16:39:19.396328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.396150 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="136e54ab-b8a1-46a4-8c4e-734d52222000" containerName="revoke-keys" Oct 30 16:39:19.396328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.396161 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="318405fd-d020-4bb2-b582-44c2517639ef" containerName="istio-proxy" Oct 30 16:39:19.396328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.396168 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ec9020-bd36-43a5-8b7a-8f753f7230b5" containerName="istio-proxy" Oct 30 16:39:19.396328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.396175 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="f680ef8d-a3ae-493c-b45f-1de23cd0b63a" containerName="main" Oct 30 16:39:19.396328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.396181 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a2dbaee-1611-4433-b37c-90487d456baf" containerName="istio-proxy" Oct 30 16:39:19.402383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.402346 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.410306 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.410277 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9"] Oct 30 16:39:19.423462 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.423423 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" containerID="cri-o://ce8a20514b06af450e34cd21b2f64d999dee33d015a385ed432060ae7c77eba7" gracePeriod=42 Oct 30 16:39:19.424954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.424933 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg"] Oct 30 16:39:19.437768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.431651 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7"] Oct 30 16:39:19.477945 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.477855 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.486942 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.484063 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7"] Oct 30 16:39:19.536526 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.536493 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:39:19.549756 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.549721 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-envoy\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.549928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.549775 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-podinfo\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.550073 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550033 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-token\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.550195 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550084 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-credential-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.550195 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550159 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istiod-ca-cert\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.550195 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550189 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-envoy\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.550330 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550216 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxrf\" (UniqueName: \"kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-kube-api-access-wqxrf\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.550363 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550337 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.550404 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550374 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkgv\" (UniqueName: \"kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-kube-api-access-qfkgv\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.550468 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550451 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-credential-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.550520 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550494 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.550608 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550589 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-data\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.550721 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550621 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-certs\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.550721 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550695 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-token\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.550855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550728 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-certs\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.550855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550793 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istiod-ca-cert\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.550995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550857 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-data\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.550995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.550890 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-podinfo\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673388 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-token\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673506 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-credential-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673545 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istiod-ca-cert\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673575 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-envoy\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673602 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxrf\" (UniqueName: \"kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-kube-api-access-wqxrf\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673659 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673703 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkgv\" (UniqueName: \"kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-kube-api-access-qfkgv\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673741 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-credential-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673786 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673869 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-data\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673908 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-certs\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673943 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-token\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.673986 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-certs\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.674041 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istiod-ca-cert\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.675337 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-certs\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.676938 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-credential-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.677298 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.677543 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-credential-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.677771 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-data\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.678580 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-certs\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.679415 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istiod-ca-cert\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.680022 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istiod-ca-cert\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.680600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.680263 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-socket\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.683675 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.682938 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-data\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.683675 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.683014 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-podinfo\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.683675 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.683070 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-envoy\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.683675 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.683116 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-podinfo\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.685382 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.685074 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-data\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.749019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.740455 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-podinfo\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.749019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.742648 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkgv\" (UniqueName: \"kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-kube-api-access-qfkgv\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.749019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.742876 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-envoy\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.749019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.743392 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-podinfo\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.749019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.743558 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxrf\" (UniqueName: \"kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-kube-api-access-wqxrf\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.749019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.743589 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-envoy\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.749019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.748622 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-token\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.749019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.748679 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-token\") pod \"e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:19.778826 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.778784 5838 generic.go:356] "Generic (PLEG): container finished" podID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerID="ce8a20514b06af450e34cd21b2f64d999dee33d015a385ed432060ae7c77eba7" exitCode=0 Oct 30 16:39:19.779269 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.779178 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" event={"ID":"887e70df-0f8a-4eb0-892f-f3ad6747a911","Type":"ContainerDied","Data":"ce8a20514b06af450e34cd21b2f64d999dee33d015a385ed432060ae7c77eba7"} Oct 30 16:39:19.830852 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.830823 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:39:19.849911 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.849881 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:19.885157 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.885125 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-dshm\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-dshm\") pod \"887e70df-0f8a-4eb0-892f-f3ad6747a911\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " Oct 30 16:39:19.885345 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.885276 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887e70df-0f8a-4eb0-892f-f3ad6747a911-kube-api-access-2fwxq\" (UniqueName: \"kubernetes.io/projected/887e70df-0f8a-4eb0-892f-f3ad6747a911-kube-api-access-2fwxq\") pod \"887e70df-0f8a-4eb0-892f-f3ad6747a911\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " Oct 30 16:39:19.885345 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.885311 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/887e70df-0f8a-4eb0-892f-f3ad6747a911-tls-certs\" (UniqueName: \"kubernetes.io/secret/887e70df-0f8a-4eb0-892f-f3ad6747a911-tls-certs\") pod \"887e70df-0f8a-4eb0-892f-f3ad6747a911\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " Oct 30 16:39:19.885460 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.885359 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-home\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-home\") pod \"887e70df-0f8a-4eb0-892f-f3ad6747a911\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " Oct 30 16:39:19.885460 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.885424 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-model-cache\") pod \"887e70df-0f8a-4eb0-892f-f3ad6747a911\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " Oct 30 16:39:19.885566 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.885510 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-tmp-dir\") pod \"887e70df-0f8a-4eb0-892f-f3ad6747a911\" (UID: \"887e70df-0f8a-4eb0-892f-f3ad6747a911\") " Oct 30 16:39:19.886063 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.886030 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-model-cache" pod "887e70df-0f8a-4eb0-892f-f3ad6747a911" (UID: "887e70df-0f8a-4eb0-892f-f3ad6747a911"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:19.886196 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.886096 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-home" pod "887e70df-0f8a-4eb0-892f-f3ad6747a911" (UID: "887e70df-0f8a-4eb0-892f-f3ad6747a911"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:19.887739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.887682 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-tmp-dir" pod "887e70df-0f8a-4eb0-892f-f3ad6747a911" (UID: "887e70df-0f8a-4eb0-892f-f3ad6747a911"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:19.891471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.891416 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887e70df-0f8a-4eb0-892f-f3ad6747a911-tls-certs" pod "887e70df-0f8a-4eb0-892f-f3ad6747a911" (UID: "887e70df-0f8a-4eb0-892f-f3ad6747a911"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:19.891620 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.891442 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-dshm" pod "887e70df-0f8a-4eb0-892f-f3ad6747a911" (UID: "887e70df-0f8a-4eb0-892f-f3ad6747a911"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:19.893457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.893415 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887e70df-0f8a-4eb0-892f-f3ad6747a911-kube-api-access-2fwxq" pod "887e70df-0f8a-4eb0-892f-f3ad6747a911" (UID: "887e70df-0f8a-4eb0-892f-f3ad6747a911"). InnerVolumeSpecName "kube-api-access-2fwxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:19.986828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.986790 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fwxq\" (UniqueName: \"kubernetes.io/projected/887e70df-0f8a-4eb0-892f-f3ad6747a911-kube-api-access-2fwxq\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:19.986828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.986826 5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887e70df-0f8a-4eb0-892f-f3ad6747a911-tls-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:19.986828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.986836 5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-home\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:19.987229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.986846 5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-model-cache\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:19.987229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.986854 5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-tmp-dir\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:19.987229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:19.986861 5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887e70df-0f8a-4eb0-892f-f3ad6747a911-dshm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:20.023711 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.023670 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:20.278057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.277925 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj"] Oct 30 16:39:20.278908 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.278887 5838 cpu_manager.go:414] "containerMap: removing container" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" Oct 30 16:39:20.279041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.278910 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" Oct 30 16:39:20.281055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.279130 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" containerName="main" Oct 30 16:39:20.290286 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:39:20.290219 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a55a46_09fd_47f5_b9dd_d725aab9bfb2.slice/crio-f6e860d632a81cabf7e8ea126a96eca3058ba9fab2d539808f51a9a5e27f1edd WatchSource:0}: Error finding container f6e860d632a81cabf7e8ea126a96eca3058ba9fab2d539808f51a9a5e27f1edd: Status 404 returned error can't find the container with id f6e860d632a81cabf7e8ea126a96eca3058ba9fab2d539808f51a9a5e27f1edd Oct 30 16:39:20.296001 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.291877 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:20.296001 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.295236 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj"] Oct 30 16:39:20.320181 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.320143 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z"] Oct 30 16:39:20.413185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.328344 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:39:20.413185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.328989 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:39:20.413185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.388289 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lzs4\" (UniqueName: \"kubernetes.io/projected/fc3546bb-7dcb-4642-bb39-411517d82ad3-kube-api-access-9lzs4\") pod \"maas-api-revoke-sub-19cb1ac3-2s9nj\" (UID: \"fc3546bb-7dcb-4642-bb39-411517d82ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:20.413185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.388430 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19cb1ac3-2s9nj\" (UID: \"fc3546bb-7dcb-4642-bb39-411517d82ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:20.413185 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.388514 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19cb1ac3-2s9nj\" (UID: \"fc3546bb-7dcb-4642-bb39-411517d82ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:20.480032 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.446752 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:39:20.485567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.481465 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:39:20.485567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.481538 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:39:20.491802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.491750 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:20.502618 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.494266 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19cb1ac3-2s9nj\" (UID: \"fc3546bb-7dcb-4642-bb39-411517d82ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:20.502928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.502911 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19cb1ac3-2s9nj\" (UID: \"fc3546bb-7dcb-4642-bb39-411517d82ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:20.548513 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.548474 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lzs4\" (UniqueName: \"kubernetes.io/projected/fc3546bb-7dcb-4642-bb39-411517d82ad3-kube-api-access-9lzs4\") pod \"maas-api-revoke-sub-19cb1ac3-2s9nj\" (UID: \"fc3546bb-7dcb-4642-bb39-411517d82ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:20.549192 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.531285 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19cb1ac3-2s9nj\" (UID: \"fc3546bb-7dcb-4642-bb39-411517d82ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:20.612803 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.612773 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z"] Oct 30 16:39:20.619402 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.619373 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7"] Oct 30 16:39:20.685008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.655807 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19cb1ac3-2s9nj\" (UID: \"fc3546bb-7dcb-4642-bb39-411517d82ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:20.758652 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.687587 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25a490c3-cct6z\" (UID: \"7940f372-7bea-44be-b44c-9a57aca6d584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:20.759319 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.759283 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6td5\" (UniqueName: \"kubernetes.io/projected/7940f372-7bea-44be-b44c-9a57aca6d584-kube-api-access-b6td5\") pod \"maas-api-revoke-sub-25a490c3-cct6z\" (UID: \"7940f372-7bea-44be-b44c-9a57aca6d584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:20.759949 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.759923 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25a490c3-cct6z\" (UID: \"7940f372-7bea-44be-b44c-9a57aca6d584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:20.821159 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.809277 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lzs4\" (UniqueName: \"kubernetes.io/projected/fc3546bb-7dcb-4642-bb39-411517d82ad3-kube-api-access-9lzs4\") pod \"maas-api-revoke-sub-19cb1ac3-2s9nj\" (UID: \"fc3546bb-7dcb-4642-bb39-411517d82ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:20.831645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.831604 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" event={"ID":"887e70df-0f8a-4eb0-892f-f3ad6747a911","Type":"ContainerDied","Data":"cc5c62c65642eebbccb5def8f19359ae7e1eb90cee2752e94eda896cbc173fb6"} Oct 30 16:39:20.831932 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.831913 5838 scope.go:122] "RemoveContainer" containerID="ce8a20514b06af450e34cd21b2f64d999dee33d015a385ed432060ae7c77eba7" Oct 30 16:39:20.832213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.832189 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57" Oct 30 16:39:20.874928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.874884 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6td5\" (UniqueName: \"kubernetes.io/projected/7940f372-7bea-44be-b44c-9a57aca6d584-kube-api-access-b6td5\") pod \"maas-api-revoke-sub-25a490c3-cct6z\" (UID: \"7940f372-7bea-44be-b44c-9a57aca6d584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:20.875328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.875299 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25a490c3-cct6z\" (UID: \"7940f372-7bea-44be-b44c-9a57aca6d584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:20.875483 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.875465 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25a490c3-cct6z\" (UID: \"7940f372-7bea-44be-b44c-9a57aca6d584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:20.876721 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.876694 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25a490c3-cct6z\" (UID: \"7940f372-7bea-44be-b44c-9a57aca6d584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:20.882478 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.882433 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" event={"ID":"32a55a46-09fd-47f5-b9dd-d725aab9bfb2","Type":"ContainerStarted","Data":"f6e860d632a81cabf7e8ea126a96eca3058ba9fab2d539808f51a9a5e27f1edd"} Oct 30 16:39:20.900416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.900338 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25a490c3-cct6z\" (UID: \"7940f372-7bea-44be-b44c-9a57aca6d584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:20.903129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.903098 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg"] Oct 30 16:39:20.914206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.914169 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6td5\" (UniqueName: \"kubernetes.io/projected/7940f372-7bea-44be-b44c-9a57aca6d584-kube-api-access-b6td5\") pod \"maas-api-revoke-sub-25a490c3-cct6z\" (UID: \"7940f372-7bea-44be-b44c-9a57aca6d584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:20.947798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:20.947757 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57"] Oct 30 16:39:20.953669 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:20.953628 5838 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod887e70df_0f8a_4eb0_892f_f3ad6747a911.slice/crio-cc5c62c65642eebbccb5def8f19359ae7e1eb90cee2752e94eda896cbc173fb6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod887e70df_0f8a_4eb0_892f_f3ad6747a911.slice\": RecentStats: unable to find data in memory cache]" Oct 30 16:39:21.066039 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:21.001616 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:21.066039 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:21.055684 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b8ab9f/premium-w4-b8ab9f-kserve-5d5f58b8d8-52p57"] Oct 30 16:39:21.242626 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:21.242599 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:21.450763 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:21.446069 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:39:21.450763 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:21.446175 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:39:21.450763 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:21.446225 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:39:22.047005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:22.043572 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" event={"ID":"fa5b1376-67d3-4de7-abce-93884b4eb2ba","Type":"ContainerStarted","Data":"7b553e36edb5ac4ef09f14e6011c5a66c00a68bc76fd044315b6b573e16f4fe3"} Oct 30 16:39:22.074999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:22.071800 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" event={"ID":"32a55a46-09fd-47f5-b9dd-d725aab9bfb2","Type":"ContainerStarted","Data":"3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763"} Oct 30 16:39:22.149997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:22.143857 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podStartSLOduration=3.143833158 podStartE2EDuration="3.143833158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:22.141886487 +0000 UTC m=+1123.890475875" watchObservedRunningTime="2026-10-30 16:39:22.143833158 +0000 UTC m=+1123.892422547" Oct 30 16:39:22.336279 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:22.321121 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:22.515052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:22.438821 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:39:22.564234 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:22.564123 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 16:39:22.565930 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:22.565883 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:39:22.593095 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:22.566166 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:22.876287 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:22.876186 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 16:39:22.876587 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:22.876557 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 16:39:22.995109 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:22.995058 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:23.037997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.030199 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="887e70df-0f8a-4eb0-892f-f3ad6747a911" path="/var/lib/kubelet/pods/887e70df-0f8a-4eb0-892f-f3ad6747a911/volumes" Oct 30 16:39:23.037997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.030960 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:23.262999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.234085 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5wtjb" podUID="9f6b4090-8bc3-4396-9f43-9150857119f8" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.17:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:23.304998 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:39:23.288335 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3546bb_7dcb_4642_bb39_411517d82ad3.slice/crio-cf1544102f75787d6315a830c9303ab9ea3ced92c28a3bd1b8f679f00dd422ab WatchSource:0}: Error finding container cf1544102f75787d6315a830c9303ab9ea3ced92c28a3bd1b8f679f00dd422ab: Status 404 returned error can't find the container with id cf1544102f75787d6315a830c9303ab9ea3ced92c28a3bd1b8f679f00dd422ab Oct 30 16:39:23.340998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.340663 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:23.394998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.374373 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:39:23.439618 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.439527 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj"] Oct 30 16:39:23.439876 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.439578 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:23.440862 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.440840 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:39:23.580198 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.564905 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:23.602190 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.601334 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:23.624998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.603735 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:39:23.795372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.795326 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" event={"ID":"fa5b1376-67d3-4de7-abce-93884b4eb2ba","Type":"ContainerStarted","Data":"882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0"} Oct 30 16:39:23.825320 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.825286 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:39:23.825663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.601369 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:39:23.853996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.826753 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:39:23.877016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.868278 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 16:39:23.877297 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.877261 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 16:39:23.951125 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:23.951070 5838 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:24.074416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.074214 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:24.407122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.367340 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:24.407122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.402231 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" podUID="a4d91ca0-9721-4ff9-8831-18a73e08b207" containerName="payload-processing" containerID="cri-o://ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0" gracePeriod=25 Oct 30 16:39:24.415395 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.415356 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 16:39:24.415671 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.415641 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 16:39:24.447408 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.445608 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:24.447408 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.447121 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" podUID="fde47769-223e-49c6-b4e9-b5c9295d8d85" containerName="payload-pre-processing" containerID="cri-o://57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885" gracePeriod=25 Oct 30 16:39:24.585203 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.579339 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podStartSLOduration=5.51730132 podStartE2EDuration="5.51730132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:24.02128749 +0000 UTC m=+1125.769876891" watchObservedRunningTime="2026-10-30 16:39:24.51730132 +0000 UTC m=+1126.265890709" Oct 30 16:39:24.611114 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.609768 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:24.611114 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.610149 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:24.789207 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.789158 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z"] Oct 30 16:39:24.797066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.796125 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:24.797066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.796196 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:24.834072 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.834022 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:24.850503 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.850370 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 16:39:24.850503 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.850433 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 16:39:24.914029 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.893441 5838 generic.go:356] "Generic (PLEG): container finished" podID="594c758e-bf83-4687-bb4f-32d89ed6ee53" containerID="2743d3299cd1aa288f8bfb861c48eb1ecc60e23e579de2584e09e7f56da4b839" exitCode=0 Oct 30 16:39:24.914029 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.893705 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" event={"ID":"594c758e-bf83-4687-bb4f-32d89ed6ee53","Type":"ContainerDied","Data":"2743d3299cd1aa288f8bfb861c48eb1ecc60e23e579de2584e09e7f56da4b839"} Oct 30 16:39:24.971232 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:24.893087 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" podUID="fde47769-223e-49c6-b4e9-b5c9295d8d85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:39:25.193213 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:39:25.193184 5838 logging.go:55] [core] [Channel #1481 SubChannel #1482]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.57:50052", ServerName: "10.134.0.57:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.57:50052: connect: connection refused" Oct 30 16:39:25.530003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:25.522004 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" podUID="36e0fb5d-ff56-4fd3-a4a6-683306972ef7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:25.545173 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:25.545117 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 16:39:25.565375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:25.557577 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 16:39:25.565375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:25.557658 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 16:39:25.579058 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:39:25.576068 5838 logging.go:55] [core] [Channel #1483 SubChannel #1484]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.56:50052", ServerName: "10.134.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.56:50052: connect: connection refused" Oct 30 16:39:25.641041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:25.621053 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 16:39:25.724248 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:25.639337 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:39:25.724248 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:25.639896 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" event={"ID":"7940f372-7bea-44be-b44c-9a57aca6d584","Type":"ContainerStarted","Data":"c5e79a6bf9bad974b757b9463da6025926d9ae6271435f495c90dedf65fda07e"} Oct 30 16:39:25.777005 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:25.766601 5838 generic.go:356] "Generic (PLEG): container finished" podID="ec27bb3d-b7dd-4e18-9afb-1a76720ae40a" containerID="5489effa2ff750e1c47ca8978d4024d02c4aa6add21b6c6801aae975c404c8ba" exitCode=0 Oct 30 16:39:25.777587 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:25.777526 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" event={"ID":"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a","Type":"ContainerDied","Data":"5489effa2ff750e1c47ca8978d4024d02c4aa6add21b6c6801aae975c404c8ba"} Oct 30 16:39:25.990958 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:25.973132 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" event={"ID":"fc3546bb-7dcb-4642-bb39-411517d82ad3","Type":"ContainerStarted","Data":"cf1544102f75787d6315a830c9303ab9ea3ced92c28a3bd1b8f679f00dd422ab"} Oct 30 16:39:26.018072 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.018022 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" podUID="594c758e-bf83-4687-bb4f-32d89ed6ee53" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.57:50052\" within 1s: context deadline exceeded" Oct 30 16:39:26.066116 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.066083 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:39:26.069882 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.069650 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 16:39:26.070132 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.070102 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 16:39:26.074987 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.074936 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq" Oct 30 16:39:26.203024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.172983 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:26.406059 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.340559 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 16:39:26.406059 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.340651 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 16:39:26.536231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.521691 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" podUID="ec27bb3d-b7dd-4e18-9afb-1a76720ae40a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.56:50052\" within 1s: context deadline exceeded" Oct 30 16:39:26.536231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.521844 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:39:26.652062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.615113 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:26.652062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.641904 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" podUID="a4d91ca0-9721-4ff9-8831-18a73e08b207" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:39:26.730692 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.730635 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:26.862570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.862104 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 16:39:26.862570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:26.862203 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 16:39:27.203996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.201480 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:27.411563 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.411530 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 16:39:27.411851 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.411809 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 16:39:27.440230 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.440194 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:39:27.440505 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.440488 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:39:27.440616 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.440604 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx"] Oct 30 16:39:27.440866 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.440845 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:39:27.484335 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.484227 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:39:27.486124 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.486094 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" podUID="33384be7-b235-48c3-9cda-dad084502291" containerName="maas-api" containerID="cri-o://513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c" gracePeriod=30 Oct 30 16:39:27.497926 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.497904 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:39:27.510659 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.510520 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8"] Oct 30 16:39:27.555228 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.542133 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594c758e-bf83-4687-bb4f-32d89ed6ee53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594c758e-bf83-4687-bb4f-32d89ed6ee53-plugins-config-volume\") pod \"594c758e-bf83-4687-bb4f-32d89ed6ee53\" (UID: \"594c758e-bf83-4687-bb4f-32d89ed6ee53\") " Oct 30 16:39:27.560025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.555865 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594c758e-bf83-4687-bb4f-32d89ed6ee53-kube-api-access-2bdbw\" (UniqueName: \"kubernetes.io/projected/594c758e-bf83-4687-bb4f-32d89ed6ee53-kube-api-access-2bdbw\") pod \"594c758e-bf83-4687-bb4f-32d89ed6ee53\" (UID: \"594c758e-bf83-4687-bb4f-32d89ed6ee53\") " Oct 30 16:39:27.582438 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.569092 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:39:27.582438 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.569774 5838 generic.go:356] "Generic (PLEG): container finished" podID="fc3546bb-7dcb-4642-bb39-411517d82ad3" containerID="11bf3aba9dce27d671983940ca7cb7affebc2d1ed64eea459154819d687aabd2" exitCode=0 Oct 30 16:39:27.582438 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.569982 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" event={"ID":"fc3546bb-7dcb-4642-bb39-411517d82ad3","Type":"ContainerDied","Data":"11bf3aba9dce27d671983940ca7cb7affebc2d1ed64eea459154819d687aabd2"} Oct 30 16:39:27.594962 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.593927 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" event={"ID":"594c758e-bf83-4687-bb4f-32d89ed6ee53","Type":"ContainerDied","Data":"d69a92e15688e70ecbd68bb3096c8b3cb53d069fa21304e0855a7771aa9fb84e"} Oct 30 16:39:27.602802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.598882 5838 scope.go:122] "RemoveContainer" containerID="2743d3299cd1aa288f8bfb861c48eb1ecc60e23e579de2584e09e7f56da4b839" Oct 30 16:39:27.602802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.594222 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2" Oct 30 16:39:27.602802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.599181 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594c758e-bf83-4687-bb4f-32d89ed6ee53-plugins-config-volume" pod "594c758e-bf83-4687-bb4f-32d89ed6ee53" (UID: "594c758e-bf83-4687-bb4f-32d89ed6ee53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:27.602802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.602251 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594c758e-bf83-4687-bb4f-32d89ed6ee53-kube-api-access-2bdbw" pod "594c758e-bf83-4687-bb4f-32d89ed6ee53" (UID: "594c758e-bf83-4687-bb4f-32d89ed6ee53"). InnerVolumeSpecName "kube-api-access-2bdbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:27.665739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.664273 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-kube-api-access-826ft\" (UniqueName: \"kubernetes.io/projected/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-kube-api-access-826ft\") pod \"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a\" (UID: \"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a\") " Oct 30 16:39:27.665739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.664404 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-plugins-config-volume\") pod \"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a\" (UID: \"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a\") " Oct 30 16:39:27.665739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.665084 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/594c758e-bf83-4687-bb4f-32d89ed6ee53-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:27.665739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.665109 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bdbw\" (UniqueName: \"kubernetes.io/projected/594c758e-bf83-4687-bb4f-32d89ed6ee53-kube-api-access-2bdbw\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:27.672645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.672600 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-plugins-config-volume" pod "ec27bb3d-b7dd-4e18-9afb-1a76720ae40a" (UID: "ec27bb3d-b7dd-4e18-9afb-1a76720ae40a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:27.695512 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.695479 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4"] Oct 30 16:39:27.704223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.704166 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b8ab9f-28vc4"] Oct 30 16:39:27.746006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.742617 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-kube-api-access-826ft" pod "ec27bb3d-b7dd-4e18-9afb-1a76720ae40a" (UID: "ec27bb3d-b7dd-4e18-9afb-1a76720ae40a"). InnerVolumeSpecName "kube-api-access-826ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:27.780995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.771143 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-826ft\" (UniqueName: \"kubernetes.io/projected/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-kube-api-access-826ft\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:27.780995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.771190 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:27.875646 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.875591 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 16:39:27.875947 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.875903 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 16:39:27.990842 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:27.989325 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" podStartSLOduration=7.989303198 podStartE2EDuration="7.989303198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:27.975004965 +0000 UTC m=+1129.723594347" watchObservedRunningTime="2026-10-30 16:39:27.989303198 +0000 UTC m=+1129.737892603" Oct 30 16:39:28.063065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.062216 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 16:39:28.063065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.062280 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 16:39:28.099125 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.096275 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2"] Oct 30 16:39:28.102194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.102032 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a7610c-dcbb484bc-blcf2"] Oct 30 16:39:28.203124 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.203088 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:39:28.285236 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.285176 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-maas-api-tls\") pod \"33384be7-b235-48c3-9cda-dad084502291\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " Oct 30 16:39:28.285384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.285257 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-metrics-certs\") pod \"33384be7-b235-48c3-9cda-dad084502291\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " Oct 30 16:39:28.285422 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.285370 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33384be7-b235-48c3-9cda-dad084502291-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33384be7-b235-48c3-9cda-dad084502291-openshift-service-ca\") pod \"33384be7-b235-48c3-9cda-dad084502291\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " Oct 30 16:39:28.285473 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.285439 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33384be7-b235-48c3-9cda-dad084502291-kube-api-access-8bplr\" (UniqueName: \"kubernetes.io/projected/33384be7-b235-48c3-9cda-dad084502291-kube-api-access-8bplr\") pod \"33384be7-b235-48c3-9cda-dad084502291\" (UID: \"33384be7-b235-48c3-9cda-dad084502291\") " Oct 30 16:39:28.286025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.285998 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33384be7-b235-48c3-9cda-dad084502291-openshift-service-ca" pod "33384be7-b235-48c3-9cda-dad084502291" (UID: "33384be7-b235-48c3-9cda-dad084502291"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:28.286448 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.286427 5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33384be7-b235-48c3-9cda-dad084502291-openshift-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:28.288078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.288055 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-maas-api-tls" pod "33384be7-b235-48c3-9cda-dad084502291" (UID: "33384be7-b235-48c3-9cda-dad084502291"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:28.288196 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.288129 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-metrics-certs" pod "33384be7-b235-48c3-9cda-dad084502291" (UID: "33384be7-b235-48c3-9cda-dad084502291"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:28.294009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.293965 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33384be7-b235-48c3-9cda-dad084502291-kube-api-access-8bplr" pod "33384be7-b235-48c3-9cda-dad084502291" (UID: "33384be7-b235-48c3-9cda-dad084502291"). InnerVolumeSpecName "kube-api-access-8bplr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:28.388097 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.387670 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-maas-api-tls\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:28.388097 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.387725 5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33384be7-b235-48c3-9cda-dad084502291-metrics-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:28.388097 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.387743 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bplr\" (UniqueName: \"kubernetes.io/projected/33384be7-b235-48c3-9cda-dad084502291-kube-api-access-8bplr\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:28.408218 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.404762 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594c758e-bf83-4687-bb4f-32d89ed6ee53" path="/var/lib/kubelet/pods/594c758e-bf83-4687-bb4f-32d89ed6ee53/volumes" Oct 30 16:39:28.408218 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.405280 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5" path="/var/lib/kubelet/pods/cc5742bf-5c5a-4054-8b2a-cb4b379cbaf5/volumes" Oct 30 16:39:28.609853 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.609819 5838 generic.go:356] "Generic (PLEG): container finished" podID="7940f372-7bea-44be-b44c-9a57aca6d584" containerID="51e245f5ea2b7226b34a90f8530e1170037bf2d6d7b225ecb0fd9511b165af9c" exitCode=0 Oct 30 16:39:28.610027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.609893 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" event={"ID":"7940f372-7bea-44be-b44c-9a57aca6d584","Type":"ContainerDied","Data":"51e245f5ea2b7226b34a90f8530e1170037bf2d6d7b225ecb0fd9511b165af9c"} Oct 30 16:39:28.611353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.611331 5838 generic.go:356] "Generic (PLEG): container finished" podID="33384be7-b235-48c3-9cda-dad084502291" containerID="513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c" exitCode=0 Oct 30 16:39:28.611445 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.611423 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" event={"ID":"33384be7-b235-48c3-9cda-dad084502291","Type":"ContainerDied","Data":"513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c"} Oct 30 16:39:28.611494 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.611455 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" event={"ID":"33384be7-b235-48c3-9cda-dad084502291","Type":"ContainerDied","Data":"6d0947ee08d1f405b0d0656a0b95e2a6de509d16e9755be1215bafff1271f7e9"} Oct 30 16:39:28.611494 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.611477 5838 scope.go:122] "RemoveContainer" containerID="513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c" Oct 30 16:39:28.611591 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.611580 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx" Oct 30 16:39:28.665758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.665670 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" Oct 30 16:39:28.666985 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.666162 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt" event={"ID":"ec27bb3d-b7dd-4e18-9afb-1a76720ae40a","Type":"ContainerDied","Data":"745b67335c8a880f189cd92d53bc4a876a93fa38a6feab5af82eb6e4ae4e8abf"} Oct 30 16:39:28.677739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.677709 5838 scope.go:122] "RemoveContainer" containerID="513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c" Oct 30 16:39:28.678508 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:28.678469 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c\": container with ID starting with 513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c not found: ID does not exist" containerID="513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c" Oct 30 16:39:28.678634 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.678523 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c"} err="failed to get container status \"513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c\": rpc error: code = NotFound desc = could not find container \"513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c\": container with ID starting with 513062e2c23282be3aea353f7c815635b5a1738bbc07fcf01a9983f4d9b5412c not found: ID does not exist" Oct 30 16:39:28.678634 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.678551 5838 scope.go:122] "RemoveContainer" containerID="5489effa2ff750e1c47ca8978d4024d02c4aa6add21b6c6801aae975c404c8ba" Oct 30 16:39:28.770368 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.770198 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx"] Oct 30 16:39:28.799648 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.799611 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a7610c-7bd87cdb76-pv2mx"] Oct 30 16:39:28.973828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.972549 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" containerID="cri-o://c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96" gracePeriod=45 Oct 30 16:39:28.973828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.972718 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" containerID="cri-o://7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1" gracePeriod=45 Oct 30 16:39:28.973828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.972843 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" containerID="cri-o://bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49" gracePeriod=45 Oct 30 16:39:28.974364 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.974339 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 16:39:28.974597 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.974490 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 16:39:28.977133 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.977082 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt"] Oct 30 16:39:28.990130 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:28.990054 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a7610c-d8f846795-6lfxt"] Oct 30 16:39:29.091186 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.091143 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 16:39:29.091377 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.091214 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 16:39:29.094818 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.094739 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9"] Oct 30 16:39:29.104531 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.104511 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn"] Oct 30 16:39:29.111152 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.111114 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf544c0-brqs9"] Oct 30 16:39:29.179858 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.171569 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d156127-l7wrn"] Oct 30 16:39:29.263994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.258179 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:29.332228 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.332101 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": dial tcp 10.134.0.97:8000: connect: connection refused" Oct 30 16:39:29.382014 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.372823 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3546bb-7dcb-4642-bb39-411517d82ad3-kube-api-access-9lzs4\" (UniqueName: \"kubernetes.io/projected/fc3546bb-7dcb-4642-bb39-411517d82ad3-kube-api-access-9lzs4\") pod \"fc3546bb-7dcb-4642-bb39-411517d82ad3\" (UID: \"fc3546bb-7dcb-4642-bb39-411517d82ad3\") " Oct 30 16:39:29.382014 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.372994 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-openshift-ca\") pod \"fc3546bb-7dcb-4642-bb39-411517d82ad3\" (UID: \"fc3546bb-7dcb-4642-bb39-411517d82ad3\") " Oct 30 16:39:29.382014 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.373074 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-xks-ca\") pod \"fc3546bb-7dcb-4642-bb39-411517d82ad3\" (UID: \"fc3546bb-7dcb-4642-bb39-411517d82ad3\") " Oct 30 16:39:29.383378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.383340 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-openshift-ca" pod "fc3546bb-7dcb-4642-bb39-411517d82ad3" (UID: "fc3546bb-7dcb-4642-bb39-411517d82ad3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:29.384428 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.384391 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-xks-ca" pod "fc3546bb-7dcb-4642-bb39-411517d82ad3" (UID: "fc3546bb-7dcb-4642-bb39-411517d82ad3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:29.385130 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.385109 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3546bb-7dcb-4642-bb39-411517d82ad3-kube-api-access-9lzs4" pod "fc3546bb-7dcb-4642-bb39-411517d82ad3" (UID: "fc3546bb-7dcb-4642-bb39-411517d82ad3"). InnerVolumeSpecName "kube-api-access-9lzs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:29.479565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.479537 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.479806 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.479790 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lzs4\" (UniqueName: \"kubernetes.io/projected/fc3546bb-7dcb-4642-bb39-411517d82ad3-kube-api-access-9lzs4\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.479937 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.479925 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc3546bb-7dcb-4642-bb39-411517d82ad3-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.600376 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.600338 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:39:29.605787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.605762 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:39:29.607701 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.607663 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:39:29.693800 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.693767 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" event={"ID":"fc3546bb-7dcb-4642-bb39-411517d82ad3","Type":"ContainerDied","Data":"cf1544102f75787d6315a830c9303ab9ea3ced92c28a3bd1b8f679f00dd422ab"} Oct 30 16:39:29.694200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.693823 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf1544102f75787d6315a830c9303ab9ea3ced92c28a3bd1b8f679f00dd422ab" Oct 30 16:39:29.694200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.693776 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj" Oct 30 16:39:29.699766 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.699737 5838 generic.go:356] "Generic (PLEG): container finished" podID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerID="c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96" exitCode=0 Oct 30 16:39:29.699995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.699949 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" Oct 30 16:39:29.700176 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.700158 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" event={"ID":"855723ad-f4a4-4857-bd8e-0174f9f7d00e","Type":"ContainerDied","Data":"c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96"} Oct 30 16:39:29.700285 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.700274 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" event={"ID":"855723ad-f4a4-4857-bd8e-0174f9f7d00e","Type":"ContainerDied","Data":"9f07733dad65d826550f95696b1b3790b05b6e853944f0327c4d891327528c57"} Oct 30 16:39:29.700355 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.700347 5838 scope.go:122] "RemoveContainer" containerID="c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96" Oct 30 16:39:29.703461 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.702488 5838 generic.go:356] "Generic (PLEG): container finished" podID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerID="bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49" exitCode=0 Oct 30 16:39:29.703461 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.702572 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" event={"ID":"d1aa2186-e285-4d17-b26a-e17f9f321266","Type":"ContainerDied","Data":"bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49"} Oct 30 16:39:29.703461 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.702609 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" event={"ID":"d1aa2186-e285-4d17-b26a-e17f9f321266","Type":"ContainerDied","Data":"9623bd85b53d0dc44c84ba2bf6d4b6eae3e93964c08ec2eb284053a278be9808"} Oct 30 16:39:29.703461 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.702740 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr" Oct 30 16:39:29.705518 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.705495 5838 generic.go:356] "Generic (PLEG): container finished" podID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerID="7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1" exitCode=0 Oct 30 16:39:29.705729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.705712 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" Oct 30 16:39:29.705857 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.705840 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" event={"ID":"281c4576-fb69-4e08-bae2-dc19a6b93eae","Type":"ContainerDied","Data":"7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1"} Oct 30 16:39:29.705923 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.705872 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" event={"ID":"281c4576-fb69-4e08-bae2-dc19a6b93eae","Type":"ContainerDied","Data":"770b582759d62a14974cdf8aeffba466f5313f0bbd9a67d13229c7e6cd953105"} Oct 30 16:39:29.778215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.778180 5838 scope.go:122] "RemoveContainer" containerID="c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96" Oct 30 16:39:29.778719 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:29.778688 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96\": container with ID starting with c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96 not found: ID does not exist" containerID="c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96" Oct 30 16:39:29.778798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.778736 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96"} err="failed to get container status \"c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96\": rpc error: code = NotFound desc = could not find container \"c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96\": container with ID starting with c5bb0587d70615b7901366ade59777753437b64fcba7f7cefb9536efd71a3f96 not found: ID does not exist" Oct 30 16:39:29.778798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.778762 5838 scope.go:122] "RemoveContainer" containerID="bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49" Oct 30 16:39:29.785188 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.785161 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-model-cache\") pod \"d1aa2186-e285-4d17-b26a-e17f9f321266\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " Oct 30 16:39:29.785572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.785557 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/281c4576-fb69-4e08-bae2-dc19a6b93eae-kube-api-access-qp862\" (UniqueName: \"kubernetes.io/projected/281c4576-fb69-4e08-bae2-dc19a6b93eae-kube-api-access-qp862\") pod \"281c4576-fb69-4e08-bae2-dc19a6b93eae\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " Oct 30 16:39:29.785944 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.785929 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/281c4576-fb69-4e08-bae2-dc19a6b93eae-tls-certs\" (UniqueName: \"kubernetes.io/secret/281c4576-fb69-4e08-bae2-dc19a6b93eae-tls-certs\") pod \"281c4576-fb69-4e08-bae2-dc19a6b93eae\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " Oct 30 16:39:29.786175 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.786158 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-model-cache\") pod \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " Oct 30 16:39:29.786563 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.786311 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-home\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-home\") pod \"d1aa2186-e285-4d17-b26a-e17f9f321266\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " Oct 30 16:39:29.786794 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.786776 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tmp-dir\") pod \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " Oct 30 16:39:29.788077 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.788049 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/855723ad-f4a4-4857-bd8e-0174f9f7d00e-kube-api-access-6w5lw\" (UniqueName: \"kubernetes.io/projected/855723ad-f4a4-4857-bd8e-0174f9f7d00e-kube-api-access-6w5lw\") pod \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " Oct 30 16:39:29.788654 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.788636 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-home\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-home\") pod \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " Oct 30 16:39:29.789162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.789135 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-tmp-dir\") pod \"281c4576-fb69-4e08-bae2-dc19a6b93eae\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " Oct 30 16:39:29.790010 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.785514 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-model-cache" pod "d1aa2186-e285-4d17-b26a-e17f9f321266" (UID: "d1aa2186-e285-4d17-b26a-e17f9f321266"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:29.790211 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.786544 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-home" pod "d1aa2186-e285-4d17-b26a-e17f9f321266" (UID: "d1aa2186-e285-4d17-b26a-e17f9f321266"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:29.790211 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.787927 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tmp-dir" pod "855723ad-f4a4-4857-bd8e-0174f9f7d00e" (UID: "855723ad-f4a4-4857-bd8e-0174f9f7d00e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:29.790211 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.789030 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-home" pod "855723ad-f4a4-4857-bd8e-0174f9f7d00e" (UID: "855723ad-f4a4-4857-bd8e-0174f9f7d00e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:29.790211 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.789893 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-tmp-dir" pod "281c4576-fb69-4e08-bae2-dc19a6b93eae" (UID: "281c4576-fb69-4e08-bae2-dc19a6b93eae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:29.790582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.790566 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-tmp-dir\") pod \"d1aa2186-e285-4d17-b26a-e17f9f321266\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " Oct 30 16:39:29.791040 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.791016 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-home\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-home\") pod \"281c4576-fb69-4e08-bae2-dc19a6b93eae\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " Oct 30 16:39:29.791475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.791452 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-dshm\") pod \"281c4576-fb69-4e08-bae2-dc19a6b93eae\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " Oct 30 16:39:29.792153 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.790942 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-tmp-dir" pod "d1aa2186-e285-4d17-b26a-e17f9f321266" (UID: "d1aa2186-e285-4d17-b26a-e17f9f321266"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:29.792304 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.791346 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-home" pod "281c4576-fb69-4e08-bae2-dc19a6b93eae" (UID: "281c4576-fb69-4e08-bae2-dc19a6b93eae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:29.792401 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.792017 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281c4576-fb69-4e08-bae2-dc19a6b93eae-kube-api-access-qp862" pod "281c4576-fb69-4e08-bae2-dc19a6b93eae" (UID: "281c4576-fb69-4e08-bae2-dc19a6b93eae"). InnerVolumeSpecName "kube-api-access-qp862". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:29.792549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.792517 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tls-certs\" (UniqueName: \"kubernetes.io/secret/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tls-certs\") pod \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " Oct 30 16:39:29.792700 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.792670 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-model-cache\") pod \"281c4576-fb69-4e08-bae2-dc19a6b93eae\" (UID: \"281c4576-fb69-4e08-bae2-dc19a6b93eae\") " Oct 30 16:39:29.792881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.792864 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1aa2186-e285-4d17-b26a-e17f9f321266-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1aa2186-e285-4d17-b26a-e17f9f321266-tls-certs\") pod \"d1aa2186-e285-4d17-b26a-e17f9f321266\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " Oct 30 16:39:29.793157 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.793141 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-dshm\") pod \"d1aa2186-e285-4d17-b26a-e17f9f321266\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " Oct 30 16:39:29.793326 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.793289 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-dshm\") pod \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\" (UID: \"855723ad-f4a4-4857-bd8e-0174f9f7d00e\") " Oct 30 16:39:29.793493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.793475 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1aa2186-e285-4d17-b26a-e17f9f321266-kube-api-access-glp6h\" (UniqueName: \"kubernetes.io/projected/d1aa2186-e285-4d17-b26a-e17f9f321266-kube-api-access-glp6h\") pod \"d1aa2186-e285-4d17-b26a-e17f9f321266\" (UID: \"d1aa2186-e285-4d17-b26a-e17f9f321266\") " Oct 30 16:39:29.794324 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.794299 5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-home\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.794447 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.794329 5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tmp-dir\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.794447 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.794342 5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-home\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.794447 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.794355 5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-tmp-dir\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.794447 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.794367 5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-tmp-dir\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.794447 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.794379 5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-home\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.794447 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.794393 5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-model-cache\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.794447 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.794406 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp862\" (UniqueName: \"kubernetes.io/projected/281c4576-fb69-4e08-bae2-dc19a6b93eae-kube-api-access-qp862\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.802696 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.802662 5838 scope.go:122] "RemoveContainer" containerID="bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49" Oct 30 16:39:29.803236 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.793027 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-model-cache" pod "281c4576-fb69-4e08-bae2-dc19a6b93eae" (UID: "281c4576-fb69-4e08-bae2-dc19a6b93eae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:29.803236 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.795399 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-model-cache" pod "855723ad-f4a4-4857-bd8e-0174f9f7d00e" (UID: "855723ad-f4a4-4857-bd8e-0174f9f7d00e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:29.803236 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.802887 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1aa2186-e285-4d17-b26a-e17f9f321266-kube-api-access-glp6h" pod "d1aa2186-e285-4d17-b26a-e17f9f321266" (UID: "d1aa2186-e285-4d17-b26a-e17f9f321266"). InnerVolumeSpecName "kube-api-access-glp6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:29.803236 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.802943 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-dshm" pod "281c4576-fb69-4e08-bae2-dc19a6b93eae" (UID: "281c4576-fb69-4e08-bae2-dc19a6b93eae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:29.804712 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:29.804641 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49\": container with ID starting with bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49 not found: ID does not exist" containerID="bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49" Oct 30 16:39:29.804712 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.804679 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49"} err="failed to get container status \"bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49\": rpc error: code = NotFound desc = could not find container \"bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49\": container with ID starting with bca913f6a63be12f1d3bf070515c904d9e4985793e4acb6116ccb7edfaad0c49 not found: ID does not exist" Oct 30 16:39:29.804804 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.804712 5838 scope.go:122] "RemoveContainer" containerID="7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1" Oct 30 16:39:29.805461 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.805249 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-dshm" pod "855723ad-f4a4-4857-bd8e-0174f9f7d00e" (UID: "855723ad-f4a4-4857-bd8e-0174f9f7d00e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:29.805461 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.805369 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/281c4576-fb69-4e08-bae2-dc19a6b93eae-tls-certs" pod "281c4576-fb69-4e08-bae2-dc19a6b93eae" (UID: "281c4576-fb69-4e08-bae2-dc19a6b93eae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:29.806951 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.806927 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855723ad-f4a4-4857-bd8e-0174f9f7d00e-kube-api-access-6w5lw" pod "855723ad-f4a4-4857-bd8e-0174f9f7d00e" (UID: "855723ad-f4a4-4857-bd8e-0174f9f7d00e"). InnerVolumeSpecName "kube-api-access-6w5lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:29.807850 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.807797 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tls-certs" pod "855723ad-f4a4-4857-bd8e-0174f9f7d00e" (UID: "855723ad-f4a4-4857-bd8e-0174f9f7d00e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:29.808712 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.808653 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1aa2186-e285-4d17-b26a-e17f9f321266-tls-certs" pod "d1aa2186-e285-4d17-b26a-e17f9f321266" (UID: "d1aa2186-e285-4d17-b26a-e17f9f321266"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:29.815711 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.815648 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-dshm" pod "d1aa2186-e285-4d17-b26a-e17f9f321266" (UID: "d1aa2186-e285-4d17-b26a-e17f9f321266"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:29.853895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.853440 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 16:39:29.853895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.853500 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 16:39:29.896572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.896532 5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/281c4576-fb69-4e08-bae2-dc19a6b93eae-tls-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.896572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.896572 5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-model-cache\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.896821 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.896586 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w5lw\" (UniqueName: \"kubernetes.io/projected/855723ad-f4a4-4857-bd8e-0174f9f7d00e-kube-api-access-6w5lw\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.896821 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.896598 5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-dshm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.896821 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.896610 5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855723ad-f4a4-4857-bd8e-0174f9f7d00e-tls-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.896821 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.896620 5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/281c4576-fb69-4e08-bae2-dc19a6b93eae-model-cache\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.896821 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.896632 5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1aa2186-e285-4d17-b26a-e17f9f321266-tls-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.896821 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.896643 5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1aa2186-e285-4d17-b26a-e17f9f321266-dshm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.896821 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.896654 5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855723ad-f4a4-4857-bd8e-0174f9f7d00e-dshm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.896821 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.896675 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glp6h\" (UniqueName: \"kubernetes.io/projected/d1aa2186-e285-4d17-b26a-e17f9f321266-kube-api-access-glp6h\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:29.970757 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.970163 5838 scope.go:122] "RemoveContainer" containerID="7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1" Oct 30 16:39:29.980167 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:29.979989 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1\": container with ID starting with 7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1 not found: ID does not exist" containerID="7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1" Oct 30 16:39:29.980167 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:29.980038 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1"} err="failed to get container status \"7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1\": rpc error: code = NotFound desc = could not find container \"7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1\": container with ID starting with 7eb26499dfed2d812058dc54365d16528db4b8cdad10e9b57d97f473ed9539e1 not found: ID does not exist" Oct 30 16:39:30.043129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.041327 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 16:39:30.043129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.041388 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 16:39:30.300770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.300733 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m"] Oct 30 16:39:30.306059 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.306028 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m"] Oct 30 16:39:30.381709 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.381685 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:30.397697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.397658 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33384be7-b235-48c3-9cda-dad084502291" path="/var/lib/kubelet/pods/33384be7-b235-48c3-9cda-dad084502291/volumes" Oct 30 16:39:30.398222 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.398199 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce677bf-1156-4ede-ab6a-83b6a9268881" path="/var/lib/kubelet/pods/3ce677bf-1156-4ede-ab6a-83b6a9268881/volumes" Oct 30 16:39:30.398664 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.398649 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" path="/var/lib/kubelet/pods/855723ad-f4a4-4857-bd8e-0174f9f7d00e/volumes" Oct 30 16:39:30.399446 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.399429 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf277455-983a-424b-bbdb-a33f3757ca8d" path="/var/lib/kubelet/pods/bf277455-983a-424b-bbdb-a33f3757ca8d/volumes" Oct 30 16:39:30.399818 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.399804 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec27bb3d-b7dd-4e18-9afb-1a76720ae40a" path="/var/lib/kubelet/pods/ec27bb3d-b7dd-4e18-9afb-1a76720ae40a/volumes" Oct 30 16:39:30.411045 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.411017 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr"] Oct 30 16:39:30.451994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.448991 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/distinct-w3-ccf5c9-kserve-6c7fb566bf-t4mjr"] Oct 30 16:39:30.451994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.449082 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/premium-w3-ccf5c9-kserve-68949c48f4-q544m" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:30.467116 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.466362 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-openshift-ca" pod "7940f372-7bea-44be-b44c-9a57aca6d584" (UID: "7940f372-7bea-44be-b44c-9a57aca6d584"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:30.467116 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.466730 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.110:8000/ready\": context deadline exceeded" Oct 30 16:39:30.467456 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.467436 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-openshift-ca\") pod \"7940f372-7bea-44be-b44c-9a57aca6d584\" (UID: \"7940f372-7bea-44be-b44c-9a57aca6d584\") " Oct 30 16:39:30.467730 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.467710 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-xks-ca\") pod \"7940f372-7bea-44be-b44c-9a57aca6d584\" (UID: \"7940f372-7bea-44be-b44c-9a57aca6d584\") " Oct 30 16:39:30.468318 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.468298 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7940f372-7bea-44be-b44c-9a57aca6d584-kube-api-access-b6td5\" (UniqueName: \"kubernetes.io/projected/7940f372-7bea-44be-b44c-9a57aca6d584-kube-api-access-b6td5\") pod \"7940f372-7bea-44be-b44c-9a57aca6d584\" (UID: \"7940f372-7bea-44be-b44c-9a57aca6d584\") " Oct 30 16:39:30.469437 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.469412 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:30.470908 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.470879 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-xks-ca" pod "7940f372-7bea-44be-b44c-9a57aca6d584" (UID: "7940f372-7bea-44be-b44c-9a57aca6d584"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:30.471494 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.471465 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7940f372-7bea-44be-b44c-9a57aca6d584-kube-api-access-b6td5" pod "7940f372-7bea-44be-b44c-9a57aca6d584" (UID: "7940f372-7bea-44be-b44c-9a57aca6d584"). InnerVolumeSpecName "kube-api-access-b6td5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:30.472992 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.472960 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s"] Oct 30 16:39:30.477926 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.477903 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ccf5c9/distinct-2-w3-ccf5c9-kserve-7d5c484c98-64l9s"] Oct 30 16:39:30.571948 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.571865 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7940f372-7bea-44be-b44c-9a57aca6d584-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:30.572158 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.572139 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6td5\" (UniqueName: \"kubernetes.io/projected/7940f372-7bea-44be-b44c-9a57aca6d584-kube-api-access-b6td5\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:30.796438 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.795233 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" event={"ID":"7940f372-7bea-44be-b44c-9a57aca6d584","Type":"ContainerDied","Data":"c5e79a6bf9bad974b757b9463da6025926d9ae6271435f495c90dedf65fda07e"} Oct 30 16:39:30.796438 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.795303 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5e79a6bf9bad974b757b9463da6025926d9ae6271435f495c90dedf65fda07e" Oct 30 16:39:30.796438 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.795394 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z" Oct 30 16:39:30.852938 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.850724 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 16:39:30.852938 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:30.850789 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 16:39:31.026910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:31.026465 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 16:39:31.026910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:31.026531 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 16:39:31.798109 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:31.798068 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:39:31.850774 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:31.850739 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 16:39:31.850940 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:31.850797 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 16:39:32.024576 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.024540 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 16:39:32.024731 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.024599 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 16:39:32.385807 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.385759 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p"] Oct 30 16:39:32.396940 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.396910 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" path="/var/lib/kubelet/pods/281c4576-fb69-4e08-bae2-dc19a6b93eae/volumes" Oct 30 16:39:32.397473 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.397456 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" path="/var/lib/kubelet/pods/d1aa2186-e285-4d17-b26a-e17f9f321266/volumes" Oct 30 16:39:32.423684 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.423648 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4"] Oct 30 16:39:32.481621 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.481582 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm"] Oct 30 16:39:32.482612 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482572 5838 cpu_manager.go:414] "containerMap: removing container" podUID="33384be7-b235-48c3-9cda-dad084502291" containerName="maas-api" Oct 30 16:39:32.482748 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482618 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33384be7-b235-48c3-9cda-dad084502291" containerName="maas-api" Oct 30 16:39:32.482803 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482754 5838 cpu_manager.go:414] "containerMap: removing container" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" Oct 30 16:39:32.482803 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482773 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" Oct 30 16:39:32.482895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482800 5838 cpu_manager.go:414] "containerMap: removing container" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" Oct 30 16:39:32.482895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482814 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" Oct 30 16:39:32.482895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482852 5838 cpu_manager.go:414] "containerMap: removing container" podUID="fc3546bb-7dcb-4642-bb39-411517d82ad3" containerName="revoke-keys" Oct 30 16:39:32.482895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482862 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3546bb-7dcb-4642-bb39-411517d82ad3" containerName="revoke-keys" Oct 30 16:39:32.482895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482882 5838 cpu_manager.go:414] "containerMap: removing container" podUID="7940f372-7bea-44be-b44c-9a57aca6d584" containerName="revoke-keys" Oct 30 16:39:32.483194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482896 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7940f372-7bea-44be-b44c-9a57aca6d584" containerName="revoke-keys" Oct 30 16:39:32.483194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482928 5838 cpu_manager.go:414] "containerMap: removing container" podUID="ec27bb3d-b7dd-4e18-9afb-1a76720ae40a" containerName="payload-pre-processing" Oct 30 16:39:32.483194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482939 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec27bb3d-b7dd-4e18-9afb-1a76720ae40a" containerName="payload-pre-processing" Oct 30 16:39:32.483194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482960 5838 cpu_manager.go:414] "containerMap: removing container" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" Oct 30 16:39:32.483194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482988 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" Oct 30 16:39:32.483194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.482999 5838 cpu_manager.go:414] "containerMap: removing container" podUID="594c758e-bf83-4687-bb4f-32d89ed6ee53" containerName="payload-processing" Oct 30 16:39:32.483194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.483006 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594c758e-bf83-4687-bb4f-32d89ed6ee53" containerName="payload-processing" Oct 30 16:39:32.483508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.483249 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="281c4576-fb69-4e08-bae2-dc19a6b93eae" containerName="main" Oct 30 16:39:32.483508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.483278 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="855723ad-f4a4-4857-bd8e-0174f9f7d00e" containerName="main" Oct 30 16:39:32.483508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.483294 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3546bb-7dcb-4642-bb39-411517d82ad3" containerName="revoke-keys" Oct 30 16:39:32.483508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.483303 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="7940f372-7bea-44be-b44c-9a57aca6d584" containerName="revoke-keys" Oct 30 16:39:32.483508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.483314 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec27bb3d-b7dd-4e18-9afb-1a76720ae40a" containerName="payload-pre-processing" Oct 30 16:39:32.483508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.483324 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1aa2186-e285-4d17-b26a-e17f9f321266" containerName="main" Oct 30 16:39:32.483508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.483333 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="594c758e-bf83-4687-bb4f-32d89ed6ee53" containerName="payload-processing" Oct 30 16:39:32.483508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.483344 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="33384be7-b235-48c3-9cda-dad084502291" containerName="maas-api" Oct 30 16:39:32.490768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.490738 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:32.527684 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.518680 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm"] Oct 30 16:39:32.527684 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.525611 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl"] Oct 30 16:39:32.532856 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.531220 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:39:32.532856 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.531241 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:39:32.532856 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.532031 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:32.542390 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.541904 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl"] Oct 30 16:39:32.554435 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.554399 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:39:32.554624 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.554576 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:39:32.584197 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.583081 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rcv8\" (UniqueName: \"kubernetes.io/projected/9ff126c2-5d26-47d2-8440-8d81421441de-kube-api-access-8rcv8\") pod \"maas-api-revoke-sub-8697d674-cn9fm\" (UID: \"9ff126c2-5d26-47d2-8440-8d81421441de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:32.584197 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.583285 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8697d674-cn9fm\" (UID: \"9ff126c2-5d26-47d2-8440-8d81421441de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:32.584197 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.583430 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8697d674-cn9fm\" (UID: \"9ff126c2-5d26-47d2-8440-8d81421441de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:32.688914 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.688813 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89b2g\" (UniqueName: \"kubernetes.io/projected/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-kube-api-access-89b2g\") pod \"maas-api-revoke-sub-ad95481c-297xl\" (UID: \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:32.697481 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.689199 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad95481c-297xl\" (UID: \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:32.697481 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.689349 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rcv8\" (UniqueName: \"kubernetes.io/projected/9ff126c2-5d26-47d2-8440-8d81421441de-kube-api-access-8rcv8\") pod \"maas-api-revoke-sub-8697d674-cn9fm\" (UID: \"9ff126c2-5d26-47d2-8440-8d81421441de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:32.697481 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.689443 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8697d674-cn9fm\" (UID: \"9ff126c2-5d26-47d2-8440-8d81421441de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:32.697481 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.689510 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8697d674-cn9fm\" (UID: \"9ff126c2-5d26-47d2-8440-8d81421441de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:32.697481 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.689547 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad95481c-297xl\" (UID: \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:32.699100 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.699053 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8697d674-cn9fm\" (UID: \"9ff126c2-5d26-47d2-8440-8d81421441de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:32.708539 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.708458 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8697d674-cn9fm\" (UID: \"9ff126c2-5d26-47d2-8440-8d81421441de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:32.731890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.731847 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rcv8\" (UniqueName: \"kubernetes.io/projected/9ff126c2-5d26-47d2-8440-8d81421441de-kube-api-access-8rcv8\") pod \"maas-api-revoke-sub-8697d674-cn9fm\" (UID: \"9ff126c2-5d26-47d2-8440-8d81421441de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:32.791264 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.791220 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89b2g\" (UniqueName: \"kubernetes.io/projected/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-kube-api-access-89b2g\") pod \"maas-api-revoke-sub-ad95481c-297xl\" (UID: \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:32.791264 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.791265 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad95481c-297xl\" (UID: \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:32.791674 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.791418 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad95481c-297xl\" (UID: \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:32.793046 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.792957 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad95481c-297xl\" (UID: \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:32.794839 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.794818 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad95481c-297xl\" (UID: \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:32.815580 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.815548 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89b2g\" (UniqueName: \"kubernetes.io/projected/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-kube-api-access-89b2g\") pod \"maas-api-revoke-sub-ad95481c-297xl\" (UID: \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:32.845013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.844944 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:32.856900 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.854450 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 16:39:32.856900 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.854513 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 16:39:32.871029 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:32.870965 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:33.025062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:33.024595 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 16:39:33.025062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:33.024651 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 16:39:33.274352 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:33.273183 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm"] Oct 30 16:39:33.421013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:33.413341 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl"] Oct 30 16:39:33.836078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:33.833313 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" event={"ID":"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3","Type":"ContainerStarted","Data":"195ca087faab9e9398df3ed5f2d4b8401404d1c2f855cdf4440964b90938a8c2"} Oct 30 16:39:33.836078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:33.835389 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" event={"ID":"9ff126c2-5d26-47d2-8440-8d81421441de","Type":"ContainerStarted","Data":"6fbbb9590d5094e357bbdbff3fbc22168b38863748011f09d17f6d11551f2a8f"} Oct 30 16:39:34.879999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:34.877383 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:34.879999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:34.879787 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" podUID="fde47769-223e-49c6-b4e9-b5c9295d8d85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:39:34.887995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:34.886181 5838 generic.go:356] "Generic (PLEG): container finished" podID="9ff126c2-5d26-47d2-8440-8d81421441de" containerID="8358b356aba757cc66ddbc5d9cceafaf009386b1213a45c6ee0251ed6b736932" exitCode=0 Oct 30 16:39:34.887995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:34.886496 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" event={"ID":"9ff126c2-5d26-47d2-8440-8d81421441de","Type":"ContainerDied","Data":"8358b356aba757cc66ddbc5d9cceafaf009386b1213a45c6ee0251ed6b736932"} Oct 30 16:39:34.907999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:34.905805 5838 generic.go:356] "Generic (PLEG): container finished" podID="7d7a25b6-d5b4-470f-b5a6-66b57f2545d3" containerID="a7092a95f28cd68277f9e06d670476102f78fea94aad183de06c993ee7c83c81" exitCode=0 Oct 30 16:39:34.907999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:34.905866 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" event={"ID":"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3","Type":"ContainerDied","Data":"a7092a95f28cd68277f9e06d670476102f78fea94aad183de06c993ee7c83c81"} Oct 30 16:39:34.921183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:34.921133 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:34.921534 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:34.921465 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:35.049554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.049518 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:35.050356 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.050331 5838 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" secret="" err="secret \"e2e-worker-w0-a7610c-openshift-default-dockercfg-8hbdj\" not found" Oct 30 16:39:35.060471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.060419 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7"] Oct 30 16:39:35.075882 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.075146 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc"] Oct 30 16:39:35.075882 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.075390 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" containerID="cri-o://7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9" gracePeriod=30 Oct 30 16:39:35.093560 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.093010 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:35.093560 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.093072 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:35.097960 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.096188 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz"] Oct 30 16:39:35.097960 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.096494 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" podUID="e20e8844-c4fd-40fb-884e-5c53aa339890" containerName="istio-proxy" containerID="cri-o://0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267" gracePeriod=30 Oct 30 16:39:35.112857 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.112472 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg"] Oct 30 16:39:35.176072 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.174862 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw"] Oct 30 16:39:35.176072 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.175142 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" containerID="cri-o://c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a" gracePeriod=30 Oct 30 16:39:35.192343 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.191833 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2"] Oct 30 16:39:35.192343 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.192080 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" podUID="9dc48445-4345-4a22-bc03-6c2108da11b6" containerName="istio-proxy" containerID="cri-o://b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05" gracePeriod=30 Oct 30 16:39:35.197390 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.196349 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v"] Oct 30 16:39:35.197390 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.196570 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerName="istio-proxy" containerID="cri-o://385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17" gracePeriod=30 Oct 30 16:39:35.704667 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.704626 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:35.874856 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.874823 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:35.876184 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.876147 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" podUID="a4d91ca0-9721-4ff9-8831-18a73e08b207" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:39:35.914992 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.914938 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" containerID="cri-o://882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0" gracePeriod=30 Oct 30 16:39:35.915486 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.914997 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" containerID="cri-o://3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763" gracePeriod=30 Oct 30 16:39:35.916248 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.916210 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:35.916928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:35.916909 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:36.098345 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.098308 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:36.228332 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.226658 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-kube-api-access-89b2g\" (UniqueName: \"kubernetes.io/projected/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-kube-api-access-89b2g\") pod \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\" (UID: \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\") " Oct 30 16:39:36.228332 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.226943 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-openshift-ca\") pod \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\" (UID: \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\") " Oct 30 16:39:36.228332 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.227148 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-xks-ca\") pod \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\" (UID: \"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3\") " Oct 30 16:39:36.228962 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.228930 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-openshift-ca" pod "7d7a25b6-d5b4-470f-b5a6-66b57f2545d3" (UID: "7d7a25b6-d5b4-470f-b5a6-66b57f2545d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:36.244751 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.239818 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-kube-api-access-89b2g" pod "7d7a25b6-d5b4-470f-b5a6-66b57f2545d3" (UID: "7d7a25b6-d5b4-470f-b5a6-66b57f2545d3"). InnerVolumeSpecName "kube-api-access-89b2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:36.244751 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.241165 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:36.253367 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.253316 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-xks-ca" pod "7d7a25b6-d5b4-470f-b5a6-66b57f2545d3" (UID: "7d7a25b6-d5b4-470f-b5a6-66b57f2545d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:36.328342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.328298 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-openshift-ca\") pod \"9ff126c2-5d26-47d2-8440-8d81421441de\" (UID: \"9ff126c2-5d26-47d2-8440-8d81421441de\") " Oct 30 16:39:36.328521 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.328505 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-xks-ca\") pod \"9ff126c2-5d26-47d2-8440-8d81421441de\" (UID: \"9ff126c2-5d26-47d2-8440-8d81421441de\") " Oct 30 16:39:36.328590 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.328556 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff126c2-5d26-47d2-8440-8d81421441de-kube-api-access-8rcv8\" (UniqueName: \"kubernetes.io/projected/9ff126c2-5d26-47d2-8440-8d81421441de-kube-api-access-8rcv8\") pod \"9ff126c2-5d26-47d2-8440-8d81421441de\" (UID: \"9ff126c2-5d26-47d2-8440-8d81421441de\") " Oct 30 16:39:36.328778 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.328747 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-openshift-ca" pod "9ff126c2-5d26-47d2-8440-8d81421441de" (UID: "9ff126c2-5d26-47d2-8440-8d81421441de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:36.329094 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.329073 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:36.329182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.329095 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:36.329182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.329110 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:36.329182 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.329121 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89b2g\" (UniqueName: \"kubernetes.io/projected/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3-kube-api-access-89b2g\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:36.330676 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.330654 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-xks-ca" pod "9ff126c2-5d26-47d2-8440-8d81421441de" (UID: "9ff126c2-5d26-47d2-8440-8d81421441de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:36.331172 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.331146 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff126c2-5d26-47d2-8440-8d81421441de-kube-api-access-8rcv8" pod "9ff126c2-5d26-47d2-8440-8d81421441de" (UID: "9ff126c2-5d26-47d2-8440-8d81421441de"). InnerVolumeSpecName "kube-api-access-8rcv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:36.430826 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.430788 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ff126c2-5d26-47d2-8440-8d81421441de-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:36.430826 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.430823 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rcv8\" (UniqueName: \"kubernetes.io/projected/9ff126c2-5d26-47d2-8440-8d81421441de-kube-api-access-8rcv8\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:36.925691 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.925663 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" Oct 30 16:39:36.925691 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.925682 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl" event={"ID":"7d7a25b6-d5b4-470f-b5a6-66b57f2545d3","Type":"ContainerDied","Data":"195ca087faab9e9398df3ed5f2d4b8401404d1c2f855cdf4440964b90938a8c2"} Oct 30 16:39:36.926223 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.925731 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="195ca087faab9e9398df3ed5f2d4b8401404d1c2f855cdf4440964b90938a8c2" Oct 30 16:39:36.927320 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.927295 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" event={"ID":"9ff126c2-5d26-47d2-8440-8d81421441de","Type":"ContainerDied","Data":"6fbbb9590d5094e357bbdbff3fbc22168b38863748011f09d17f6d11551f2a8f"} Oct 30 16:39:36.927430 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.927334 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fbbb9590d5094e357bbdbff3fbc22168b38863748011f09d17f6d11551f2a8f" Oct 30 16:39:36.927430 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:36.927302 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm" Oct 30 16:39:37.032850 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:37.032808 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29"] Oct 30 16:39:37.037774 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:37.037742 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6827378d-4lf29"] Oct 30 16:39:37.386255 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:37.386206 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" podUID="7203fda6-f9e6-4d6f-96da-0514b72e7ca7" containerName="payload-processing" containerID="cri-o://e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba" gracePeriod=25 Oct 30 16:39:37.424009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:37.423939 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" podUID="0795bf8a-e9c9-440f-97a1-d2fdaa47f97f" containerName="payload-pre-processing" containerID="cri-o://1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163" gracePeriod=25 Oct 30 16:39:37.893243 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:37.893203 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" podUID="7203fda6-f9e6-4d6f-96da-0514b72e7ca7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:39:38.396412 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:38.396382 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c" path="/var/lib/kubelet/pods/3538f9ff-1ab1-4ac3-8fca-b79dc3106a2c/volumes" Oct 30 16:39:38.718633 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:38.718542 5838 patch_prober.go:29] interesting pod/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 16:39:38.718633 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:38.718615 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:39:40.401206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.401173 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:39:40.464502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.461955 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:39:40.505412 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.505365 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-credential-socket\") pod \"340ac92b-1be2-4337-a547-c0c0d239ca96\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " Oct 30 16:39:40.505598 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.505441 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-envoy\") pod \"340ac92b-1be2-4337-a547-c0c0d239ca96\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " Oct 30 16:39:40.505598 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.505494 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-socket\") pod \"340ac92b-1be2-4337-a547-c0c0d239ca96\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " Oct 30 16:39:40.505598 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.505575 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/340ac92b-1be2-4337-a547-c0c0d239ca96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340ac92b-1be2-4337-a547-c0c0d239ca96-istiod-ca-cert\") pod \"340ac92b-1be2-4337-a547-c0c0d239ca96\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " Oct 30 16:39:40.505773 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.505615 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-token\" (UniqueName: \"kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-token\") pod \"340ac92b-1be2-4337-a547-c0c0d239ca96\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " Oct 30 16:39:40.505773 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.505703 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-podinfo\") pod \"340ac92b-1be2-4337-a547-c0c0d239ca96\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " Oct 30 16:39:40.505773 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.505741 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-data\") pod \"340ac92b-1be2-4337-a547-c0c0d239ca96\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " Oct 30 16:39:40.505928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.505783 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-kube-api-access-fq5j2\" (UniqueName: \"kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-kube-api-access-fq5j2\") pod \"340ac92b-1be2-4337-a547-c0c0d239ca96\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " Oct 30 16:39:40.505928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.505844 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-certs\") pod \"340ac92b-1be2-4337-a547-c0c0d239ca96\" (UID: \"340ac92b-1be2-4337-a547-c0c0d239ca96\") " Oct 30 16:39:40.506553 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.506410 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-credential-socket" pod "340ac92b-1be2-4337-a547-c0c0d239ca96" (UID: "340ac92b-1be2-4337-a547-c0c0d239ca96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.506676 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.506646 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-data" pod "340ac92b-1be2-4337-a547-c0c0d239ca96" (UID: "340ac92b-1be2-4337-a547-c0c0d239ca96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.506868 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.506842 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-socket" pod "340ac92b-1be2-4337-a547-c0c0d239ca96" (UID: "340ac92b-1be2-4337-a547-c0c0d239ca96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.507015 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.506887 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-certs" pod "340ac92b-1be2-4337-a547-c0c0d239ca96" (UID: "340ac92b-1be2-4337-a547-c0c0d239ca96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.507644 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.507607 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340ac92b-1be2-4337-a547-c0c0d239ca96-istiod-ca-cert" pod "340ac92b-1be2-4337-a547-c0c0d239ca96" (UID: "340ac92b-1be2-4337-a547-c0c0d239ca96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:40.514572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.513711 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-podinfo" pod "340ac92b-1be2-4337-a547-c0c0d239ca96" (UID: "340ac92b-1be2-4337-a547-c0c0d239ca96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:40.514572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.514495 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-token" pod "340ac92b-1be2-4337-a547-c0c0d239ca96" (UID: "340ac92b-1be2-4337-a547-c0c0d239ca96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:40.514903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.514866 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-kube-api-access-fq5j2" pod "340ac92b-1be2-4337-a547-c0c0d239ca96" (UID: "340ac92b-1be2-4337-a547-c0c0d239ca96"). InnerVolumeSpecName "kube-api-access-fq5j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:40.515639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.515617 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-envoy" pod "340ac92b-1be2-4337-a547-c0c0d239ca96" (UID: "340ac92b-1be2-4337-a547-c0c0d239ca96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.580789 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.580759 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:39:40.607582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.606882 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20e8844-c4fd-40fb-884e-5c53aa339890-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20e8844-c4fd-40fb-884e-5c53aa339890-istiod-ca-cert\") pod \"e20e8844-c4fd-40fb-884e-5c53aa339890\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " Oct 30 16:39:40.607582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.606993 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-credential-socket\") pod \"e20e8844-c4fd-40fb-884e-5c53aa339890\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " Oct 30 16:39:40.608087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.608044 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20e8844-c4fd-40fb-884e-5c53aa339890-istiod-ca-cert" pod "e20e8844-c4fd-40fb-884e-5c53aa339890" (UID: "e20e8844-c4fd-40fb-884e-5c53aa339890"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:40.608550 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.608513 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-credential-socket" pod "e20e8844-c4fd-40fb-884e-5c53aa339890" (UID: "e20e8844-c4fd-40fb-884e-5c53aa339890"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.617075 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.617043 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-socket\") pod \"e20e8844-c4fd-40fb-884e-5c53aa339890\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " Oct 30 16:39:40.617252 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.617135 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-envoy\") pod \"e20e8844-c4fd-40fb-884e-5c53aa339890\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " Oct 30 16:39:40.617252 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.617217 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-kube-api-access-lgjbv\" (UniqueName: \"kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-kube-api-access-lgjbv\") pod \"e20e8844-c4fd-40fb-884e-5c53aa339890\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " Oct 30 16:39:40.617436 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.617285 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-certs\") pod \"e20e8844-c4fd-40fb-884e-5c53aa339890\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " Oct 30 16:39:40.617436 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.617322 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-token\" (UniqueName: \"kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-token\") pod \"e20e8844-c4fd-40fb-884e-5c53aa339890\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " Oct 30 16:39:40.617436 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.617369 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-podinfo\") pod \"e20e8844-c4fd-40fb-884e-5c53aa339890\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " Oct 30 16:39:40.617618 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.617441 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-data\") pod \"e20e8844-c4fd-40fb-884e-5c53aa339890\" (UID: \"e20e8844-c4fd-40fb-884e-5c53aa339890\") " Oct 30 16:39:40.618012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.617965 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-socket" pod "e20e8844-c4fd-40fb-884e-5c53aa339890" (UID: "e20e8844-c4fd-40fb-884e-5c53aa339890"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.618012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.618001 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-data" pod "e20e8844-c4fd-40fb-884e-5c53aa339890" (UID: "e20e8844-c4fd-40fb-884e-5c53aa339890"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.618301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.618278 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.618395 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.618381 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.618469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.618458 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.618544 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.618532 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.618644 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.618633 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340ac92b-1be2-4337-a547-c0c0d239ca96-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.618951 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.618725 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.619103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.619090 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20e8844-c4fd-40fb-884e-5c53aa339890-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.619189 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.619177 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.619279 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.619265 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.619363 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.619351 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.619460 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.619446 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq5j2\" (UniqueName: \"kubernetes.io/projected/340ac92b-1be2-4337-a547-c0c0d239ca96-kube-api-access-fq5j2\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.619875 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.619518 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-certs" pod "e20e8844-c4fd-40fb-884e-5c53aa339890" (UID: "e20e8844-c4fd-40fb-884e-5c53aa339890"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.619875 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.619539 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.619875 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.619554 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340ac92b-1be2-4337-a547-c0c0d239ca96-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.620609 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.620586 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-podinfo" pod "e20e8844-c4fd-40fb-884e-5c53aa339890" (UID: "e20e8844-c4fd-40fb-884e-5c53aa339890"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:40.623500 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.623471 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-token" pod "e20e8844-c4fd-40fb-884e-5c53aa339890" (UID: "e20e8844-c4fd-40fb-884e-5c53aa339890"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:40.623639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.623532 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-envoy" pod "e20e8844-c4fd-40fb-884e-5c53aa339890" (UID: "e20e8844-c4fd-40fb-884e-5c53aa339890"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.623827 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.623804 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-kube-api-access-lgjbv" pod "e20e8844-c4fd-40fb-884e-5c53aa339890" (UID: "e20e8844-c4fd-40fb-884e-5c53aa339890"). InnerVolumeSpecName "kube-api-access-lgjbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:40.628764 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.628740 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:39:40.637673 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.637647 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:39:40.720069 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720034 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-socket\") pod \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " Oct 30 16:39:40.720268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720086 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-podinfo\") pod \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " Oct 30 16:39:40.720268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720110 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istiod-ca-cert\") pod \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " Oct 30 16:39:40.720268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720129 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-data\") pod \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " Oct 30 16:39:40.720268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720154 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-envoy\") pod \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " Oct 30 16:39:40.720268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720197 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-credential-socket\") pod \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " Oct 30 16:39:40.720268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720235 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-socket\") pod \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " Oct 30 16:39:40.720268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720271 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-envoy\") pod \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " Oct 30 16:39:40.720647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720316 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-certs\") pod \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " Oct 30 16:39:40.720647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720359 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-certs\") pod \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " Oct 30 16:39:40.720647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720365 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-socket" pod "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" (UID: "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.720647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720379 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-token\" (UniqueName: \"kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-token\") pod \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " Oct 30 16:39:40.720647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720488 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-kube-api-access-8z8mz\" (UniqueName: \"kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-kube-api-access-8z8mz\") pod \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " Oct 30 16:39:40.720647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720590 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-data\") pod \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\" (UID: \"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe\") " Oct 30 16:39:40.720944 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720711 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-kube-api-access-6fzpd\" (UniqueName: \"kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-kube-api-access-6fzpd\") pod \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " Oct 30 16:39:40.720944 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720758 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istiod-ca-cert\") pod \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " Oct 30 16:39:40.720944 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720808 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-credential-socket\") pod \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " Oct 30 16:39:40.720944 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720930 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-podinfo\") pod \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " Oct 30 16:39:40.721180 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.721050 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-token\" (UniqueName: \"kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-token\") pod \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\" (UID: \"439bafeb-d93c-46f1-a33f-1a1ffcd1d776\") " Oct 30 16:39:40.721667 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.721643 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.721785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.721669 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.721785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.721686 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgjbv\" (UniqueName: \"kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-kube-api-access-lgjbv\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.721785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.721701 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20e8844-c4fd-40fb-884e-5c53aa339890-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.721785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.721710 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.721785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.721718 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20e8844-c4fd-40fb-884e-5c53aa339890-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.724538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720592 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-socket" pod "439bafeb-d93c-46f1-a33f-1a1ffcd1d776" (UID: "439bafeb-d93c-46f1-a33f-1a1ffcd1d776"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.724538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.720771 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-data" pod "439bafeb-d93c-46f1-a33f-1a1ffcd1d776" (UID: "439bafeb-d93c-46f1-a33f-1a1ffcd1d776"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.724538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.722420 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istiod-ca-cert" pod "439bafeb-d93c-46f1-a33f-1a1ffcd1d776" (UID: "439bafeb-d93c-46f1-a33f-1a1ffcd1d776"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:40.724538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.721182 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-certs" pod "439bafeb-d93c-46f1-a33f-1a1ffcd1d776" (UID: "439bafeb-d93c-46f1-a33f-1a1ffcd1d776"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.724538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.722135 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-credential-socket" pod "439bafeb-d93c-46f1-a33f-1a1ffcd1d776" (UID: "439bafeb-d93c-46f1-a33f-1a1ffcd1d776"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.724538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.722996 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-credential-socket" pod "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" (UID: "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.724538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.723057 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-data" pod "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" (UID: "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.724538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.723313 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istiod-ca-cert" pod "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" (UID: "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:40.724538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.723807 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-certs" pod "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" (UID: "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.724538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.724268 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-envoy" pod "439bafeb-d93c-46f1-a33f-1a1ffcd1d776" (UID: "439bafeb-d93c-46f1-a33f-1a1ffcd1d776"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.725387 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.725348 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-kube-api-access-6fzpd" pod "439bafeb-d93c-46f1-a33f-1a1ffcd1d776" (UID: "439bafeb-d93c-46f1-a33f-1a1ffcd1d776"). InnerVolumeSpecName "kube-api-access-6fzpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:40.726141 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.726089 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-kube-api-access-8z8mz" pod "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" (UID: "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe"). InnerVolumeSpecName "kube-api-access-8z8mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:40.726420 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.726395 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-envoy" pod "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" (UID: "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.726886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.726862 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-token" pod "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" (UID: "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:40.727380 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.727356 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-token" pod "439bafeb-d93c-46f1-a33f-1a1ffcd1d776" (UID: "439bafeb-d93c-46f1-a33f-1a1ffcd1d776"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:40.727797 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.727771 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-podinfo" pod "439bafeb-d93c-46f1-a33f-1a1ffcd1d776" (UID: "439bafeb-d93c-46f1-a33f-1a1ffcd1d776"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:40.727910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.727854 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-podinfo" pod "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" (UID: "0d6f8643-fe14-47eb-81f1-bd89bd12bdbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:40.822742 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.822702 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-certs\") pod \"9dc48445-4345-4a22-bc03-6c2108da11b6\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " Oct 30 16:39:40.822742 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.822752 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-data\") pod \"9dc48445-4345-4a22-bc03-6c2108da11b6\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " Oct 30 16:39:40.822965 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.822790 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc48445-4345-4a22-bc03-6c2108da11b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc48445-4345-4a22-bc03-6c2108da11b6-istiod-ca-cert\") pod \"9dc48445-4345-4a22-bc03-6c2108da11b6\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " Oct 30 16:39:40.822965 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.822862 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-podinfo\") pod \"9dc48445-4345-4a22-bc03-6c2108da11b6\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " Oct 30 16:39:40.823098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823031 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-envoy\") pod \"9dc48445-4345-4a22-bc03-6c2108da11b6\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " Oct 30 16:39:40.823098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823039 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-certs" pod "9dc48445-4345-4a22-bc03-6c2108da11b6" (UID: "9dc48445-4345-4a22-bc03-6c2108da11b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.823098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823090 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-socket\") pod \"9dc48445-4345-4a22-bc03-6c2108da11b6\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " Oct 30 16:39:40.823231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823120 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-data" pod "9dc48445-4345-4a22-bc03-6c2108da11b6" (UID: "9dc48445-4345-4a22-bc03-6c2108da11b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.823231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823128 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-token\" (UniqueName: \"kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-token\") pod \"9dc48445-4345-4a22-bc03-6c2108da11b6\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " Oct 30 16:39:40.823342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823237 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-kube-api-access-qx8pd\" (UniqueName: \"kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-kube-api-access-qx8pd\") pod \"9dc48445-4345-4a22-bc03-6c2108da11b6\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " Oct 30 16:39:40.823342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823278 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc48445-4345-4a22-bc03-6c2108da11b6-istiod-ca-cert" pod "9dc48445-4345-4a22-bc03-6c2108da11b6" (UID: "9dc48445-4345-4a22-bc03-6c2108da11b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:40.823342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823293 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-credential-socket\") pod \"9dc48445-4345-4a22-bc03-6c2108da11b6\" (UID: \"9dc48445-4345-4a22-bc03-6c2108da11b6\") " Oct 30 16:39:40.823495 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823406 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-socket" pod "9dc48445-4345-4a22-bc03-6c2108da11b6" (UID: "9dc48445-4345-4a22-bc03-6c2108da11b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.823672 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823649 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-credential-socket" pod "9dc48445-4345-4a22-bc03-6c2108da11b6" (UID: "9dc48445-4345-4a22-bc03-6c2108da11b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.823924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823901 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824035 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823929 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824035 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823947 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824035 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.823960 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824035 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824013 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc48445-4345-4a22-bc03-6c2108da11b6-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824035 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824027 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824042 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824056 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824071 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824089 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824103 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824116 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824129 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824142 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z8mz\" (UniqueName: \"kubernetes.io/projected/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-kube-api-access-8z8mz\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824156 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824170 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824184 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fzpd\" (UniqueName: \"kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-kube-api-access-6fzpd\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824199 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824216 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824229 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824243 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.824333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.824256 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439bafeb-d93c-46f1-a33f-1a1ffcd1d776-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.825302 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.825281 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-envoy" pod "9dc48445-4345-4a22-bc03-6c2108da11b6" (UID: "9dc48445-4345-4a22-bc03-6c2108da11b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:40.825302 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.825290 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-token" pod "9dc48445-4345-4a22-bc03-6c2108da11b6" (UID: "9dc48445-4345-4a22-bc03-6c2108da11b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:40.825427 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.825291 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-podinfo" pod "9dc48445-4345-4a22-bc03-6c2108da11b6" (UID: "9dc48445-4345-4a22-bc03-6c2108da11b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:40.825629 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.825596 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-kube-api-access-qx8pd" pod "9dc48445-4345-4a22-bc03-6c2108da11b6" (UID: "9dc48445-4345-4a22-bc03-6c2108da11b6"). InnerVolumeSpecName "kube-api-access-qx8pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:40.926112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.925949 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.926112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.926041 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.926112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.926058 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.926112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.926071 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx8pd\" (UniqueName: \"kubernetes.io/projected/9dc48445-4345-4a22-bc03-6c2108da11b6-kube-api-access-qx8pd\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:40.968433 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.968400 5838 generic.go:356] "Generic (PLEG): container finished" podID="e20e8844-c4fd-40fb-884e-5c53aa339890" containerID="0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267" exitCode=0 Oct 30 16:39:40.968617 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.968442 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" Oct 30 16:39:40.968617 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.968475 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" event={"ID":"e20e8844-c4fd-40fb-884e-5c53aa339890","Type":"ContainerDied","Data":"0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267"} Oct 30 16:39:40.968617 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.968532 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz" event={"ID":"e20e8844-c4fd-40fb-884e-5c53aa339890","Type":"ContainerDied","Data":"c093eb777661cc64ec49b73a73bf9e943b874a3a373ef29c6a725b1bb0267ad6"} Oct 30 16:39:40.968617 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.968554 5838 scope.go:122] "RemoveContainer" containerID="0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267" Oct 30 16:39:40.970662 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.970631 5838 generic.go:356] "Generic (PLEG): container finished" podID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerID="c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a" exitCode=0 Oct 30 16:39:40.970785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.970687 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" Oct 30 16:39:40.970785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.970706 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" event={"ID":"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe","Type":"ContainerDied","Data":"c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a"} Oct 30 16:39:40.970785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.970754 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw" event={"ID":"0d6f8643-fe14-47eb-81f1-bd89bd12bdbe","Type":"ContainerDied","Data":"c6ac3c2eee2110dd7d8febae50c2acdc200358da24d00d6884453a6284e78c13"} Oct 30 16:39:40.972167 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.972142 5838 generic.go:356] "Generic (PLEG): container finished" podID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerID="385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17" exitCode=0 Oct 30 16:39:40.972283 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.972186 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" Oct 30 16:39:40.972283 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.972224 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" event={"ID":"439bafeb-d93c-46f1-a33f-1a1ffcd1d776","Type":"ContainerDied","Data":"385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17"} Oct 30 16:39:40.972283 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.972272 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v" event={"ID":"439bafeb-d93c-46f1-a33f-1a1ffcd1d776","Type":"ContainerDied","Data":"ec00d6d46cd6c70f5a9c69017daf3af6e72869fbf95f7a6f4149360f3a10ef1a"} Oct 30 16:39:40.973708 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.973685 5838 generic.go:356] "Generic (PLEG): container finished" podID="9dc48445-4345-4a22-bc03-6c2108da11b6" containerID="b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05" exitCode=0 Oct 30 16:39:40.973794 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.973721 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" Oct 30 16:39:40.973794 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.973759 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" event={"ID":"9dc48445-4345-4a22-bc03-6c2108da11b6","Type":"ContainerDied","Data":"b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05"} Oct 30 16:39:40.973901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.973800 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2" event={"ID":"9dc48445-4345-4a22-bc03-6c2108da11b6","Type":"ContainerDied","Data":"2d6f33f4d154be08ce16c111a6ea497456d7d9f8fa1085ed4da2cd2bea2b9978"} Oct 30 16:39:40.975885 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.975862 5838 generic.go:356] "Generic (PLEG): container finished" podID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerID="7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9" exitCode=0 Oct 30 16:39:40.975991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.975925 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" Oct 30 16:39:40.975991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.975938 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" event={"ID":"340ac92b-1be2-4337-a547-c0c0d239ca96","Type":"ContainerDied","Data":"7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9"} Oct 30 16:39:40.976118 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:40.976009 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc" event={"ID":"340ac92b-1be2-4337-a547-c0c0d239ca96","Type":"ContainerDied","Data":"071f0f84be996dfe67b4ec6cd574b5330aff5337a5fd4e6d6b0786b5569f3b52"} Oct 30 16:39:41.058579 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.058552 5838 scope.go:122] "RemoveContainer" containerID="0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267" Oct 30 16:39:41.058964 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:41.058941 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267\": container with ID starting with 0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267 not found: ID does not exist" containerID="0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267" Oct 30 16:39:41.058964 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.058990 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267"} err="failed to get container status \"0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267\": rpc error: code = NotFound desc = could not find container \"0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267\": container with ID starting with 0a213b7f7f6e6d2b247def9b2dc1da68c5a3780b50cd141ebc4b82e429613267 not found: ID does not exist" Oct 30 16:39:41.059227 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.059012 5838 scope.go:122] "RemoveContainer" containerID="c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a" Oct 30 16:39:41.078274 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.078162 5838 scope.go:122] "RemoveContainer" containerID="73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81" Oct 30 16:39:41.097578 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.097473 5838 scope.go:122] "RemoveContainer" containerID="c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a" Oct 30 16:39:41.098220 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:41.098078 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a\": container with ID starting with c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a not found: ID does not exist" containerID="c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a" Oct 30 16:39:41.098220 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.098116 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a"} err="failed to get container status \"c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a\": rpc error: code = NotFound desc = could not find container \"c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a\": container with ID starting with c4d48bfc9e6dcc8fe596189a17d8b25021ec82ed1364ba206fab2890208dda0a not found: ID does not exist" Oct 30 16:39:41.098220 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.098139 5838 scope.go:122] "RemoveContainer" containerID="73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81" Oct 30 16:39:41.098839 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:41.098685 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81\": container with ID starting with 73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81 not found: ID does not exist" containerID="73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81" Oct 30 16:39:41.098839 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.098715 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81"} err="failed to get container status \"73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81\": rpc error: code = NotFound desc = could not find container \"73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81\": container with ID starting with 73e557f4bddd3975e0d3d537eced70f686d82b323a1accf68322193c71118f81 not found: ID does not exist" Oct 30 16:39:41.098839 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.098738 5838 scope.go:122] "RemoveContainer" containerID="385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17" Oct 30 16:39:41.138022 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.124895 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw"] Oct 30 16:39:41.138022 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.131716 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-b79zw"] Oct 30 16:39:41.147762 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.147733 5838 scope.go:122] "RemoveContainer" containerID="385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17" Oct 30 16:39:41.150121 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:41.148234 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17\": container with ID starting with 385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17 not found: ID does not exist" containerID="385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17" Oct 30 16:39:41.150121 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.148277 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17"} err="failed to get container status \"385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17\": rpc error: code = NotFound desc = could not find container \"385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17\": container with ID starting with 385a2386888f5fe4cd8c50a5719c45ad56a890cd8cc3eb8c4b665b0367a4ef17 not found: ID does not exist" Oct 30 16:39:41.150121 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.148303 5838 scope.go:122] "RemoveContainer" containerID="b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05" Oct 30 16:39:41.158660 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.158604 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v"] Oct 30 16:39:41.169855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.169815 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-v727v"] Oct 30 16:39:41.188244 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.178823 5838 scope.go:122] "RemoveContainer" containerID="b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05" Oct 30 16:39:41.197850 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:41.196207 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05\": container with ID starting with b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05 not found: ID does not exist" containerID="b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05" Oct 30 16:39:41.197850 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.196254 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05"} err="failed to get container status \"b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05\": rpc error: code = NotFound desc = could not find container \"b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05\": container with ID starting with b8977d492ea1bea9a4a5096ea3481395af5d338a21046838891aab9396d4cb05 not found: ID does not exist" Oct 30 16:39:41.197850 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.196282 5838 scope.go:122] "RemoveContainer" containerID="7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9" Oct 30 16:39:41.207640 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.207613 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2"] Oct 30 16:39:41.216644 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.216616 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b8ab9f-openshift-default-7796766fb5-8rwz2"] Oct 30 16:39:41.237080 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.237051 5838 scope.go:122] "RemoveContainer" containerID="84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c" Oct 30 16:39:41.241711 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.241588 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:41.250132 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.250102 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc"] Oct 30 16:39:41.252806 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.252788 5838 scope.go:122] "RemoveContainer" containerID="7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9" Oct 30 16:39:41.253323 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:41.253192 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9\": container with ID starting with 7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9 not found: ID does not exist" containerID="7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9" Oct 30 16:39:41.253323 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.253231 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9"} err="failed to get container status \"7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9\": rpc error: code = NotFound desc = could not find container \"7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9\": container with ID starting with 7779d46638863fe4e50a12d4b6f15e39a6dbd70a2b711874e641d3286f6d78e9 not found: ID does not exist" Oct 30 16:39:41.253323 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.253254 5838 scope.go:122] "RemoveContainer" containerID="84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c" Oct 30 16:39:41.253839 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:41.253810 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c\": container with ID starting with 84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c not found: ID does not exist" containerID="84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c" Oct 30 16:39:41.253955 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.253847 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c"} err="failed to get container status \"84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c\": rpc error: code = NotFound desc = could not find container \"84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c\": container with ID starting with 84d4950e79cb94b66fb789b3daacd41ceb99abbc385aae1172e2d39d0359e62c not found: ID does not exist" Oct 30 16:39:41.256689 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.256668 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-82vnc"] Oct 30 16:39:41.291473 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.291439 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz"] Oct 30 16:39:41.353995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.341467 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-socket\") pod \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " Oct 30 16:39:41.388989 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.347553 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-socket" pod "32a55a46-09fd-47f5-b9dd-d725aab9bfb2" (UID: "32a55a46-09fd-47f5-b9dd-d725aab9bfb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:41.388989 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.383791 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-envoy\") pod \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " Oct 30 16:39:41.388989 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.383859 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-data\") pod \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " Oct 30 16:39:41.388989 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.384251 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-data" pod "32a55a46-09fd-47f5-b9dd-d725aab9bfb2" (UID: "32a55a46-09fd-47f5-b9dd-d725aab9bfb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:41.388989 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.384724 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-token\" (UniqueName: \"kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-token\") pod \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " Oct 30 16:39:41.388989 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.385131 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-credential-socket\") pod \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " Oct 30 16:39:41.388989 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.388871 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-envoy" pod "32a55a46-09fd-47f5-b9dd-d725aab9bfb2" (UID: "32a55a46-09fd-47f5-b9dd-d725aab9bfb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:41.393197 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.393142 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-token" pod "32a55a46-09fd-47f5-b9dd-d725aab9bfb2" (UID: "32a55a46-09fd-47f5-b9dd-d725aab9bfb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:41.393557 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.393458 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-credential-socket" pod "32a55a46-09fd-47f5-b9dd-d725aab9bfb2" (UID: "32a55a46-09fd-47f5-b9dd-d725aab9bfb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:41.395864 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.394929 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-podinfo\") pod \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " Oct 30 16:39:41.395864 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.394962 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-kube-api-access-wqxrf\" (UniqueName: \"kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-kube-api-access-wqxrf\") pod \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " Oct 30 16:39:41.395864 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.395086 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istiod-ca-cert\") pod \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " Oct 30 16:39:41.395864 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.395138 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-certs\") pod \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\" (UID: \"32a55a46-09fd-47f5-b9dd-d725aab9bfb2\") " Oct 30 16:39:41.395864 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.395616 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.395864 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.395641 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.395864 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.395655 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.395864 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.395668 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.395864 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.395680 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.396657 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.396497 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-certs" pod "32a55a46-09fd-47f5-b9dd-d725aab9bfb2" (UID: "32a55a46-09fd-47f5-b9dd-d725aab9bfb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:41.396961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.396909 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istiod-ca-cert" pod "32a55a46-09fd-47f5-b9dd-d725aab9bfb2" (UID: "32a55a46-09fd-47f5-b9dd-d725aab9bfb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:41.403990 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.403859 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:41.417086 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.417030 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-kube-api-access-wqxrf" pod "32a55a46-09fd-47f5-b9dd-d725aab9bfb2" (UID: "32a55a46-09fd-47f5-b9dd-d725aab9bfb2"). InnerVolumeSpecName "kube-api-access-wqxrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:41.432154 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.432075 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-podinfo" pod "32a55a46-09fd-47f5-b9dd-d725aab9bfb2" (UID: "32a55a46-09fd-47f5-b9dd-d725aab9bfb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:41.454599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.454562 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-s65xz"] Oct 30 16:39:41.496082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496041 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-podinfo\") pod \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " Oct 30 16:39:41.496264 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496097 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-socket\") pod \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " Oct 30 16:39:41.496264 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496136 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-data\") pod \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " Oct 30 16:39:41.496264 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496195 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-token\" (UniqueName: \"kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-token\") pod \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " Oct 30 16:39:41.496264 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496220 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-kube-api-access-qfkgv\" (UniqueName: \"kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-kube-api-access-qfkgv\") pod \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " Oct 30 16:39:41.496469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496273 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istiod-ca-cert\") pod \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " Oct 30 16:39:41.496469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496309 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-envoy\") pod \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " Oct 30 16:39:41.496469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496328 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-credential-socket\") pod \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " Oct 30 16:39:41.496469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496345 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-certs\") pod \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\" (UID: \"fa5b1376-67d3-4de7-abce-93884b4eb2ba\") " Oct 30 16:39:41.496469 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496444 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-socket" pod "fa5b1376-67d3-4de7-abce-93884b4eb2ba" (UID: "fa5b1376-67d3-4de7-abce-93884b4eb2ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:41.496752 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496727 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-certs" pod "fa5b1376-67d3-4de7-abce-93884b4eb2ba" (UID: "fa5b1376-67d3-4de7-abce-93884b4eb2ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:41.496812 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496768 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-credential-socket" pod "fa5b1376-67d3-4de7-abce-93884b4eb2ba" (UID: "fa5b1376-67d3-4de7-abce-93884b4eb2ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:41.496956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.496936 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-data" pod "fa5b1376-67d3-4de7-abce-93884b4eb2ba" (UID: "fa5b1376-67d3-4de7-abce-93884b4eb2ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:41.497163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.497141 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istiod-ca-cert" pod "fa5b1376-67d3-4de7-abce-93884b4eb2ba" (UID: "fa5b1376-67d3-4de7-abce-93884b4eb2ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:41.497442 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.497420 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.497639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.497449 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.497639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.497467 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.497639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.497481 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqxrf\" (UniqueName: \"kubernetes.io/projected/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-kube-api-access-wqxrf\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.497639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.497495 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.497639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.497510 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.497639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.497524 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a55a46-09fd-47f5-b9dd-d725aab9bfb2-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.498629 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.498605 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-podinfo" pod "fa5b1376-67d3-4de7-abce-93884b4eb2ba" (UID: "fa5b1376-67d3-4de7-abce-93884b4eb2ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:41.498871 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.498855 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-envoy" pod "fa5b1376-67d3-4de7-abce-93884b4eb2ba" (UID: "fa5b1376-67d3-4de7-abce-93884b4eb2ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:41.499201 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.499186 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-token" pod "fa5b1376-67d3-4de7-abce-93884b4eb2ba" (UID: "fa5b1376-67d3-4de7-abce-93884b4eb2ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:41.499268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.499256 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-kube-api-access-qfkgv" pod "fa5b1376-67d3-4de7-abce-93884b4eb2ba" (UID: "fa5b1376-67d3-4de7-abce-93884b4eb2ba"). InnerVolumeSpecName "kube-api-access-qfkgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:41.684668 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.684577 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.684889 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.684869 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfkgv\" (UniqueName: \"kubernetes.io/projected/fa5b1376-67d3-4de7-abce-93884b4eb2ba-kube-api-access-qfkgv\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.690402 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.689055 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.690402 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.689088 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.690402 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.689105 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.690402 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.689118 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5b1376-67d3-4de7-abce-93884b4eb2ba-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:41.868062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.868027 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd"] Oct 30 16:39:41.873077 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.873040 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ccf5c9-nj9gd"] Oct 30 16:39:41.995003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.994953 5838 generic.go:356] "Generic (PLEG): container finished" podID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerID="3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763" exitCode=0 Oct 30 16:39:41.995330 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.995303 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" event={"ID":"32a55a46-09fd-47f5-b9dd-d725aab9bfb2","Type":"ContainerDied","Data":"3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763"} Oct 30 16:39:41.995501 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.995482 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" event={"ID":"32a55a46-09fd-47f5-b9dd-d725aab9bfb2","Type":"ContainerDied","Data":"f6e860d632a81cabf7e8ea126a96eca3058ba9fab2d539808f51a9a5e27f1edd"} Oct 30 16:39:41.995635 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.995621 5838 scope.go:122] "RemoveContainer" containerID="3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763" Oct 30 16:39:41.995865 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:41.995848 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7" Oct 30 16:39:42.004062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.004034 5838 generic.go:356] "Generic (PLEG): container finished" podID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerID="882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0" exitCode=0 Oct 30 16:39:42.004279 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.004258 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" event={"ID":"fa5b1376-67d3-4de7-abce-93884b4eb2ba","Type":"ContainerDied","Data":"882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0"} Oct 30 16:39:42.004423 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.004407 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" event={"ID":"fa5b1376-67d3-4de7-abce-93884b4eb2ba","Type":"ContainerDied","Data":"7b553e36edb5ac4ef09f14e6011c5a66c00a68bc76fd044315b6b573e16f4fe3"} Oct 30 16:39:42.004619 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.004603 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg" Oct 30 16:39:42.023106 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.023069 5838 scope.go:122] "RemoveContainer" containerID="3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763" Oct 30 16:39:42.023485 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:42.023457 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763\": container with ID starting with 3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763 not found: ID does not exist" containerID="3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763" Oct 30 16:39:42.023558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.023500 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763"} err="failed to get container status \"3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763\": rpc error: code = NotFound desc = could not find container \"3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763\": container with ID starting with 3f9f72030ec71e311786c5bb38587b74af59ea354955b823a535c951d1acc763 not found: ID does not exist" Oct 30 16:39:42.023558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.023526 5838 scope.go:122] "RemoveContainer" containerID="882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0" Oct 30 16:39:42.049335 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.049269 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6"] Oct 30 16:39:42.054645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.054615 5838 scope.go:122] "RemoveContainer" containerID="882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0" Oct 30 16:39:42.055133 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:42.055096 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0\": container with ID starting with 882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0 not found: ID does not exist" containerID="882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0" Oct 30 16:39:42.055328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.055300 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0"} err="failed to get container status \"882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0\": rpc error: code = NotFound desc = could not find container \"882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0\": container with ID starting with 882c1f7ae5eee6a8cc89c65fbea7dfc07b8f45b2aa791bc162ccc2d523fc78d0 not found: ID does not exist" Oct 30 16:39:42.056410 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.056387 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb"] Oct 30 16:39:42.063826 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.063802 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc8fa13d-scgr6"] Oct 30 16:39:42.070889 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.070866 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf303e92-gtvbb"] Oct 30 16:39:42.078744 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.078719 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7"] Oct 30 16:39:42.085797 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.085768 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-bcch7"] Oct 30 16:39:42.138786 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.138730 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg"] Oct 30 16:39:42.139460 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.139439 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a7610c-openshift-default-6457bd6bfc-wlqhg"] Oct 30 16:39:42.422991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.422889 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" path="/var/lib/kubelet/pods/0d6f8643-fe14-47eb-81f1-bd89bd12bdbe/volumes" Oct 30 16:39:42.434961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.434929 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302d2f6e-678e-48b2-810d-1bdfb9d36342" path="/var/lib/kubelet/pods/302d2f6e-678e-48b2-810d-1bdfb9d36342/volumes" Oct 30 16:39:42.435930 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.435905 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" path="/var/lib/kubelet/pods/32a55a46-09fd-47f5-b9dd-d725aab9bfb2/volumes" Oct 30 16:39:42.437899 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.437878 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" path="/var/lib/kubelet/pods/340ac92b-1be2-4337-a547-c0c0d239ca96/volumes" Oct 30 16:39:42.438842 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.438821 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" path="/var/lib/kubelet/pods/439bafeb-d93c-46f1-a33f-1a1ffcd1d776/volumes" Oct 30 16:39:42.448565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.448535 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="906e3afd-5693-4dde-8904-5ef536742cbb" path="/var/lib/kubelet/pods/906e3afd-5693-4dde-8904-5ef536742cbb/volumes" Oct 30 16:39:42.449655 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.449624 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc48445-4345-4a22-bc03-6c2108da11b6" path="/var/lib/kubelet/pods/9dc48445-4345-4a22-bc03-6c2108da11b6/volumes" Oct 30 16:39:42.450597 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.450578 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab729303-64bc-42b4-8f35-fa9d5aa5ca66" path="/var/lib/kubelet/pods/ab729303-64bc-42b4-8f35-fa9d5aa5ca66/volumes" Oct 30 16:39:42.452178 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.452158 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e20e8844-c4fd-40fb-884e-5c53aa339890" path="/var/lib/kubelet/pods/e20e8844-c4fd-40fb-884e-5c53aa339890/volumes" Oct 30 16:39:42.452937 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:42.452918 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" path="/var/lib/kubelet/pods/fa5b1376-67d3-4de7-abce-93884b4eb2ba/volumes" Oct 30 16:39:43.699781 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.699737 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" containerID="cri-o://232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7" gracePeriod=45 Oct 30 16:39:43.967422 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.967352 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:39:43.984648 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.984616 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e579322a-2a82-46bd-8959-e75ee51bb57d-kube-api-access-kn8tl\" (UniqueName: \"kubernetes.io/projected/e579322a-2a82-46bd-8959-e75ee51bb57d-kube-api-access-kn8tl\") pod \"e579322a-2a82-46bd-8959-e75ee51bb57d\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " Oct 30 16:39:43.984808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.984670 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-home\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-home\") pod \"e579322a-2a82-46bd-8959-e75ee51bb57d\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " Oct 30 16:39:43.984874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.984856 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-dshm\") pod \"e579322a-2a82-46bd-8959-e75ee51bb57d\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " Oct 30 16:39:43.984920 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.984897 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e579322a-2a82-46bd-8959-e75ee51bb57d-tls-certs\" (UniqueName: \"kubernetes.io/secret/e579322a-2a82-46bd-8959-e75ee51bb57d-tls-certs\") pod \"e579322a-2a82-46bd-8959-e75ee51bb57d\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " Oct 30 16:39:43.984986 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.984933 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-model-cache\") pod \"e579322a-2a82-46bd-8959-e75ee51bb57d\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " Oct 30 16:39:43.984986 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.984952 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-home" pod "e579322a-2a82-46bd-8959-e75ee51bb57d" (UID: "e579322a-2a82-46bd-8959-e75ee51bb57d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:43.985088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.985046 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-tmp-dir\") pod \"e579322a-2a82-46bd-8959-e75ee51bb57d\" (UID: \"e579322a-2a82-46bd-8959-e75ee51bb57d\") " Oct 30 16:39:43.985193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.985165 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-model-cache" pod "e579322a-2a82-46bd-8959-e75ee51bb57d" (UID: "e579322a-2a82-46bd-8959-e75ee51bb57d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:43.985331 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.985304 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-tmp-dir" pod "e579322a-2a82-46bd-8959-e75ee51bb57d" (UID: "e579322a-2a82-46bd-8959-e75ee51bb57d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:43.985515 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.985496 5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-model-cache\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:43.985594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.985522 5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-tmp-dir\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:43.985594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.985536 5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-home\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:43.987054 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.987028 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e579322a-2a82-46bd-8959-e75ee51bb57d-tls-certs" pod "e579322a-2a82-46bd-8959-e75ee51bb57d" (UID: "e579322a-2a82-46bd-8959-e75ee51bb57d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:43.987195 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.987094 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-dshm" pod "e579322a-2a82-46bd-8959-e75ee51bb57d" (UID: "e579322a-2a82-46bd-8959-e75ee51bb57d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:43.987195 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:43.987145 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e579322a-2a82-46bd-8959-e75ee51bb57d-kube-api-access-kn8tl" pod "e579322a-2a82-46bd-8959-e75ee51bb57d" (UID: "e579322a-2a82-46bd-8959-e75ee51bb57d"). InnerVolumeSpecName "kube-api-access-kn8tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:44.041363 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.041328 5838 generic.go:356] "Generic (PLEG): container finished" podID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerID="232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7" exitCode=0 Oct 30 16:39:44.041545 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.041372 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" Oct 30 16:39:44.041545 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.041416 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" event={"ID":"e579322a-2a82-46bd-8959-e75ee51bb57d","Type":"ContainerDied","Data":"232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7"} Oct 30 16:39:44.041545 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.041472 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8" event={"ID":"e579322a-2a82-46bd-8959-e75ee51bb57d","Type":"ContainerDied","Data":"6dbaaedba3fe4ad3597420fba1a8e4aa8aac9edc3e5f297c496640845869fea3"} Oct 30 16:39:44.041545 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.041498 5838 scope.go:122] "RemoveContainer" containerID="232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7" Oct 30 16:39:44.054495 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.054469 5838 scope.go:122] "RemoveContainer" containerID="232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7" Oct 30 16:39:44.055057 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:44.055027 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7\": container with ID starting with 232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7 not found: ID does not exist" containerID="232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7" Oct 30 16:39:44.055158 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.055070 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7"} err="failed to get container status \"232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7\": rpc error: code = NotFound desc = could not find container \"232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7\": container with ID starting with 232daf2473c409a32b1925d72b9ea709f264f45eae27c8bff31b1691d7ac52b7 not found: ID does not exist" Oct 30 16:39:44.086462 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.086426 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn8tl\" (UniqueName: \"kubernetes.io/projected/e579322a-2a82-46bd-8959-e75ee51bb57d-kube-api-access-kn8tl\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:44.086462 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.086456 5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e579322a-2a82-46bd-8959-e75ee51bb57d-dshm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:44.086462 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.086467 5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e579322a-2a82-46bd-8959-e75ee51bb57d-tls-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:44.100152 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.099750 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8"] Oct 30 16:39:44.105725 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.105697 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b55521/premium-w1-b55521-kserve-6f84795d55-nkkq8"] Oct 30 16:39:44.395481 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.395445 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" path="/var/lib/kubelet/pods/e579322a-2a82-46bd-8959-e75ee51bb57d/volumes" Oct 30 16:39:44.959369 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.959339 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" Oct 30 16:39:44.961918 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:44.961893 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" Oct 30 16:39:45.113845 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.111290 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d91ca0-9721-4ff9-8831-18a73e08b207-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d91ca0-9721-4ff9-8831-18a73e08b207-plugins-config-volume\") pod \"a4d91ca0-9721-4ff9-8831-18a73e08b207\" (UID: \"a4d91ca0-9721-4ff9-8831-18a73e08b207\") " Oct 30 16:39:45.113845 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.111362 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde47769-223e-49c6-b4e9-b5c9295d8d85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde47769-223e-49c6-b4e9-b5c9295d8d85-plugins-config-volume\") pod \"fde47769-223e-49c6-b4e9-b5c9295d8d85\" (UID: \"fde47769-223e-49c6-b4e9-b5c9295d8d85\") " Oct 30 16:39:45.113845 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.111455 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d91ca0-9721-4ff9-8831-18a73e08b207-kube-api-access-pz9wj\" (UniqueName: \"kubernetes.io/projected/a4d91ca0-9721-4ff9-8831-18a73e08b207-kube-api-access-pz9wj\") pod \"a4d91ca0-9721-4ff9-8831-18a73e08b207\" (UID: \"a4d91ca0-9721-4ff9-8831-18a73e08b207\") " Oct 30 16:39:45.113845 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.112934 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde47769-223e-49c6-b4e9-b5c9295d8d85-kube-api-access-hmk8f\" (UniqueName: \"kubernetes.io/projected/fde47769-223e-49c6-b4e9-b5c9295d8d85-kube-api-access-hmk8f\") pod \"fde47769-223e-49c6-b4e9-b5c9295d8d85\" (UID: \"fde47769-223e-49c6-b4e9-b5c9295d8d85\") " Oct 30 16:39:45.114565 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.114514 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d91ca0-9721-4ff9-8831-18a73e08b207-plugins-config-volume" pod "a4d91ca0-9721-4ff9-8831-18a73e08b207" (UID: "a4d91ca0-9721-4ff9-8831-18a73e08b207"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:45.115743 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.115162 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde47769-223e-49c6-b4e9-b5c9295d8d85-plugins-config-volume" pod "fde47769-223e-49c6-b4e9-b5c9295d8d85" (UID: "fde47769-223e-49c6-b4e9-b5c9295d8d85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:45.115929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.115895 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d91ca0-9721-4ff9-8831-18a73e08b207-kube-api-access-pz9wj" pod "a4d91ca0-9721-4ff9-8831-18a73e08b207" (UID: "a4d91ca0-9721-4ff9-8831-18a73e08b207"). InnerVolumeSpecName "kube-api-access-pz9wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:45.117650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.117628 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde47769-223e-49c6-b4e9-b5c9295d8d85-kube-api-access-hmk8f" pod "fde47769-223e-49c6-b4e9-b5c9295d8d85" (UID: "fde47769-223e-49c6-b4e9-b5c9295d8d85"). InnerVolumeSpecName "kube-api-access-hmk8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:45.139174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.130123 5838 generic.go:356] "Generic (PLEG): container finished" podID="fde47769-223e-49c6-b4e9-b5c9295d8d85" containerID="57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885" exitCode=0 Oct 30 16:39:45.139174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.130308 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" Oct 30 16:39:45.139174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.133149 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" event={"ID":"fde47769-223e-49c6-b4e9-b5c9295d8d85","Type":"ContainerDied","Data":"57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885"} Oct 30 16:39:45.139174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.133227 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" event={"ID":"fde47769-223e-49c6-b4e9-b5c9295d8d85","Type":"ContainerDied","Data":"58bfb7657681b4870bbf11729432268f787bef3bfb1acae029656db9163af352"} Oct 30 16:39:45.139174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.133275 5838 scope.go:122] "RemoveContainer" containerID="57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885" Oct 30 16:39:45.139174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.137453 5838 generic.go:356] "Generic (PLEG): container finished" podID="a4d91ca0-9721-4ff9-8831-18a73e08b207" containerID="ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0" exitCode=0 Oct 30 16:39:45.139174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.137501 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" event={"ID":"a4d91ca0-9721-4ff9-8831-18a73e08b207","Type":"ContainerDied","Data":"ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0"} Oct 30 16:39:45.139174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.137548 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" event={"ID":"a4d91ca0-9721-4ff9-8831-18a73e08b207","Type":"ContainerDied","Data":"8d1c4e59b2996c9c7232dcd9b55c86cd3243b560b1e41ef41d57e2e22cc1044b"} Oct 30 16:39:45.139174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.137627 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr" Oct 30 16:39:45.163219 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.163185 5838 scope.go:122] "RemoveContainer" containerID="57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885" Oct 30 16:39:45.163580 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:45.163561 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885\": container with ID starting with 57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885 not found: ID does not exist" containerID="57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885" Oct 30 16:39:45.163659 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.163593 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885"} err="failed to get container status \"57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885\": rpc error: code = NotFound desc = could not find container \"57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885\": container with ID starting with 57ca00f9d071bd63d591d4372d29c5c69db786ae4ee10edbe2aac7a51c551885 not found: ID does not exist" Oct 30 16:39:45.163659 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.163617 5838 scope.go:122] "RemoveContainer" containerID="ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0" Oct 30 16:39:45.174246 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.174227 5838 scope.go:122] "RemoveContainer" containerID="ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0" Oct 30 16:39:45.174563 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:45.174533 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0\": container with ID starting with ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0 not found: ID does not exist" containerID="ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0" Oct 30 16:39:45.174627 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.174574 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0"} err="failed to get container status \"ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0\": rpc error: code = NotFound desc = could not find container \"ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0\": container with ID starting with ada187b0d58532e2da908f0a694a3378a9b6944e0b6fc09cbdc4d0d5331022f0 not found: ID does not exist" Oct 30 16:39:45.208407 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.208325 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr"] Oct 30 16:39:45.213058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.213031 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-ccf5c9-674cd7b595-zbclr"] Oct 30 16:39:45.214664 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.214647 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz9wj\" (UniqueName: \"kubernetes.io/projected/a4d91ca0-9721-4ff9-8831-18a73e08b207-kube-api-access-pz9wj\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:45.214710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.214667 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmk8f\" (UniqueName: \"kubernetes.io/projected/fde47769-223e-49c6-b4e9-b5c9295d8d85-kube-api-access-hmk8f\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:45.214710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.214678 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4d91ca0-9721-4ff9-8831-18a73e08b207-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:45.214710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.214687 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde47769-223e-49c6-b4e9-b5c9295d8d85-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:45.246420 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.246390 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9"] Oct 30 16:39:45.250643 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.250616 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9"] Oct 30 16:39:45.878704 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:45.878656 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-ccf5c9-fbcd94fbc-b2fk9" podUID="fde47769-223e-49c6-b4e9-b5c9295d8d85" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.47:50052\" within 1s: context deadline exceeded" Oct 30 16:39:45.878864 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:39:45.878794 5838 logging.go:55] [core] [Channel #1518 SubChannel #1519]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.47:50052", ServerName: "10.134.0.47:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.47:50052: operation was canceled" Oct 30 16:39:46.395440 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:46.395407 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d91ca0-9721-4ff9-8831-18a73e08b207" path="/var/lib/kubelet/pods/a4d91ca0-9721-4ff9-8831-18a73e08b207/volumes" Oct 30 16:39:46.395861 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:46.395846 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde47769-223e-49c6-b4e9-b5c9295d8d85" path="/var/lib/kubelet/pods/fde47769-223e-49c6-b4e9-b5c9295d8d85/volumes" Oct 30 16:39:46.882187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:46.882148 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" podUID="0795bf8a-e9c9-440f-97a1-d2fdaa47f97f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:39:47.893897 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:47.893849 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" podUID="7203fda6-f9e6-4d6f-96da-0514b72e7ca7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:39:48.540074 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:48.540038 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:39:49.059339 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:49.059282 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t"] Oct 30 16:39:49.059894 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:49.059528 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerName="istio-proxy" containerID="cri-o://c65f96cb2c08540b928d3fedbb077a96413a748c15e5ce1e1ca73c3fb9292794" gracePeriod=30 Oct 30 16:39:49.070373 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:49.070337 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv"] Oct 30 16:39:49.074894 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:49.074866 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7"] Oct 30 16:39:49.075225 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:49.075191 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" podUID="39dd14bb-4cf4-4016-b2b1-59519d96fda7" containerName="istio-proxy" containerID="cri-o://434f9907ef5aedf1f52567f9d1e0335a29ae3bc6582f401c39a98e4f4c0afcc2" gracePeriod=30 Oct 30 16:39:49.079900 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:49.079874 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25ab8ece-mdgfv"] Oct 30 16:39:49.083927 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:49.083902 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px"] Oct 30 16:39:49.084125 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:49.084101 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" podUID="dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" containerName="istio-proxy" containerID="cri-o://553b8b7b7e7299ffa15795a6ddf1011fc0fd5f06ffc3c92941eea6e6e316ffe6" gracePeriod=30 Oct 30 16:39:49.102305 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:49.102273 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf"] Oct 30 16:39:49.102504 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:49.102483 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" podUID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" containerName="istio-proxy" containerID="cri-o://f8e03bf279a58db19495cecf07019a8e99d7c23180af956907f2b935b9429f6f" gracePeriod=30 Oct 30 16:39:50.395227 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:50.395187 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb757212-c7f6-47e0-9557-e45e7a630846" path="/var/lib/kubelet/pods/fb757212-c7f6-47e0-9557-e45e7a630846/volumes" Oct 30 16:39:53.029954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:53.029917 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8"] Oct 30 16:39:53.040721 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:53.040691 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d6f232-blnv8"] Oct 30 16:39:53.388810 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:53.388720 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d"] Oct 30 16:39:53.389057 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:53.389019 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" podUID="63ad2f35-f82b-43d6-b068-163cc3192b26" containerName="istio-proxy" containerID="cri-o://b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d" gracePeriod=30 Oct 30 16:39:53.406521 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:53.406488 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg"] Oct 30 16:39:53.406768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:53.406720 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerName="istio-proxy" containerID="cri-o://49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e" gracePeriod=30 Oct 30 16:39:53.413146 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:53.413118 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj"] Oct 30 16:39:53.413379 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:53.413348 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" podUID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" containerName="istio-proxy" containerID="cri-o://506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192" gracePeriod=30 Oct 30 16:39:53.418165 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:53.418140 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd"] Oct 30 16:39:53.418343 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:53.418323 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" containerID="cri-o://cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6" gracePeriod=30 Oct 30 16:39:54.314592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.314034 5838 generic.go:356] "Generic (PLEG): container finished" podID="39dd14bb-4cf4-4016-b2b1-59519d96fda7" containerID="434f9907ef5aedf1f52567f9d1e0335a29ae3bc6582f401c39a98e4f4c0afcc2" exitCode=0 Oct 30 16:39:54.314592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.314131 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" event={"ID":"39dd14bb-4cf4-4016-b2b1-59519d96fda7","Type":"ContainerDied","Data":"434f9907ef5aedf1f52567f9d1e0335a29ae3bc6582f401c39a98e4f4c0afcc2"} Oct 30 16:39:54.318735 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.318580 5838 generic.go:356] "Generic (PLEG): container finished" podID="dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" containerID="553b8b7b7e7299ffa15795a6ddf1011fc0fd5f06ffc3c92941eea6e6e316ffe6" exitCode=0 Oct 30 16:39:54.318735 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.318656 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" event={"ID":"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1","Type":"ContainerDied","Data":"553b8b7b7e7299ffa15795a6ddf1011fc0fd5f06ffc3c92941eea6e6e316ffe6"} Oct 30 16:39:54.321434 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.321213 5838 generic.go:356] "Generic (PLEG): container finished" podID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" containerID="f8e03bf279a58db19495cecf07019a8e99d7c23180af956907f2b935b9429f6f" exitCode=0 Oct 30 16:39:54.321434 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.321367 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" event={"ID":"13833b5b-ab9b-4a32-811b-5eb07645cfa9","Type":"ContainerDied","Data":"f8e03bf279a58db19495cecf07019a8e99d7c23180af956907f2b935b9429f6f"} Oct 30 16:39:54.324409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.324191 5838 generic.go:356] "Generic (PLEG): container finished" podID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerID="c65f96cb2c08540b928d3fedbb077a96413a748c15e5ce1e1ca73c3fb9292794" exitCode=0 Oct 30 16:39:54.324409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.324344 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" event={"ID":"d62ed47a-62cf-46a3-84f4-c06c2a13e22c","Type":"ContainerDied","Data":"c65f96cb2c08540b928d3fedbb077a96413a748c15e5ce1e1ca73c3fb9292794"} Oct 30 16:39:54.389675 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.389643 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:39:54.397245 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.397212 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeaff967-55d3-45bb-b745-ac0a320ece42" path="/var/lib/kubelet/pods/eeaff967-55d3-45bb-b745-ac0a320ece42/volumes" Oct 30 16:39:54.431255 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.431222 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:39:54.449681 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.449609 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istiod-ca-cert\") pod \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " Oct 30 16:39:54.449681 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.449676 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-data\") pod \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " Oct 30 16:39:54.450632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.449776 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-certs\") pod \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " Oct 30 16:39:54.450632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.449875 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-token\" (UniqueName: \"kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-token\") pod \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " Oct 30 16:39:54.450632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.449910 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-credential-socket\") pod \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " Oct 30 16:39:54.450632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.449944 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-socket\") pod \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " Oct 30 16:39:54.450632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.449954 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-data" pod "d62ed47a-62cf-46a3-84f4-c06c2a13e22c" (UID: "d62ed47a-62cf-46a3-84f4-c06c2a13e22c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.450632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.450018 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-envoy\") pod \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " Oct 30 16:39:54.450632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.450107 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-podinfo\") pod \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " Oct 30 16:39:54.450632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.450133 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-certs" pod "d62ed47a-62cf-46a3-84f4-c06c2a13e22c" (UID: "d62ed47a-62cf-46a3-84f4-c06c2a13e22c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.450632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.450187 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-kube-api-access-xtzgz\" (UniqueName: \"kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-kube-api-access-xtzgz\") pod \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\" (UID: \"d62ed47a-62cf-46a3-84f4-c06c2a13e22c\") " Oct 30 16:39:54.450632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.450402 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-socket" pod "d62ed47a-62cf-46a3-84f4-c06c2a13e22c" (UID: "d62ed47a-62cf-46a3-84f4-c06c2a13e22c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.450632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.450372 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:39:54.450632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.450438 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-credential-socket" pod "d62ed47a-62cf-46a3-84f4-c06c2a13e22c" (UID: "d62ed47a-62cf-46a3-84f4-c06c2a13e22c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.451343 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.451321 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.451847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.451771 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.451847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.451793 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.451847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.451808 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.451847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.451499 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istiod-ca-cert" pod "d62ed47a-62cf-46a3-84f4-c06c2a13e22c" (UID: "d62ed47a-62cf-46a3-84f4-c06c2a13e22c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:54.453404 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.453150 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-kube-api-access-xtzgz" pod "d62ed47a-62cf-46a3-84f4-c06c2a13e22c" (UID: "d62ed47a-62cf-46a3-84f4-c06c2a13e22c"). InnerVolumeSpecName "kube-api-access-xtzgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:54.454714 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.453415 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-token" pod "d62ed47a-62cf-46a3-84f4-c06c2a13e22c" (UID: "d62ed47a-62cf-46a3-84f4-c06c2a13e22c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:54.457208 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.457174 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-envoy" pod "d62ed47a-62cf-46a3-84f4-c06c2a13e22c" (UID: "d62ed47a-62cf-46a3-84f4-c06c2a13e22c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.457309 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.457239 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-podinfo" pod "d62ed47a-62cf-46a3-84f4-c06c2a13e22c" (UID: "d62ed47a-62cf-46a3-84f4-c06c2a13e22c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:54.472598 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.472574 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:39:54.552433 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552397 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-credential-socket\") pod \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " Oct 30 16:39:54.552433 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552439 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-data\") pod \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " Oct 30 16:39:54.552678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552467 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-envoy\") pod \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " Oct 30 16:39:54.552678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552497 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-certs\") pod \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " Oct 30 16:39:54.552678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552524 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-data\") pod \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " Oct 30 16:39:54.552678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552553 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istiod-ca-cert\") pod \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " Oct 30 16:39:54.552678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552580 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-podinfo\") pod \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " Oct 30 16:39:54.552678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552617 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-socket\") pod \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " Oct 30 16:39:54.552678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552642 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-kube-api-access-hqfls\" (UniqueName: \"kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-kube-api-access-hqfls\") pod \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " Oct 30 16:39:54.552678 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552674 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-token\" (UniqueName: \"kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-token\") pod \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552702 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-credential-socket" pod "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" (UID: "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552709 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-kube-api-access-8fmd6\" (UniqueName: \"kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-kube-api-access-8fmd6\") pod \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552769 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-envoy\") pod \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552804 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-credential-socket\") pod \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552874 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-token\" (UniqueName: \"kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-token\") pod \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552902 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-kube-api-access-lxs2x\" (UniqueName: \"kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-kube-api-access-lxs2x\") pod \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552940 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istiod-ca-cert\") pod \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.552991 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-socket\") pod \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553024 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-envoy\") pod \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553053 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-podinfo\") pod \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553095 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-credential-socket\") pod \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553139 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-podinfo\") pod \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553221 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-certs\") pod \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\" (UID: \"13833b5b-ab9b-4a32-811b-5eb07645cfa9\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553281 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istiod-ca-cert\") pod \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553320 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-data\") pod \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553327 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istiod-ca-cert" pod "13833b5b-ab9b-4a32-811b-5eb07645cfa9" (UID: "13833b5b-ab9b-4a32-811b-5eb07645cfa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553360 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-socket\") pod \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553397 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-token\" (UniqueName: \"kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-token\") pod \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\" (UID: \"39dd14bb-4cf4-4016-b2b1-59519d96fda7\") " Oct 30 16:39:54.553493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553459 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-certs\") pod \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\" (UID: \"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1\") " Oct 30 16:39:54.554516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553683 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-data" pod "39dd14bb-4cf4-4016-b2b1-59519d96fda7" (UID: "39dd14bb-4cf4-4016-b2b1-59519d96fda7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.554516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.553889 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-certs" pod "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" (UID: "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.554516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.554074 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.554516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.554094 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.554516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.554108 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.554516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.554121 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.554516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.554136 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.554516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.554150 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.554516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.554163 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.554516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.554178 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.554516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.554192 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtzgz\" (UniqueName: \"kubernetes.io/projected/d62ed47a-62cf-46a3-84f4-c06c2a13e22c-kube-api-access-xtzgz\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.555095 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.554549 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-socket" pod "13833b5b-ab9b-4a32-811b-5eb07645cfa9" (UID: "13833b5b-ab9b-4a32-811b-5eb07645cfa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.555425 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.555390 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-kube-api-access-8fmd6" pod "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" (UID: "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1"). InnerVolumeSpecName "kube-api-access-8fmd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:54.556284 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.556024 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-token" pod "13833b5b-ab9b-4a32-811b-5eb07645cfa9" (UID: "13833b5b-ab9b-4a32-811b-5eb07645cfa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:54.556383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.556296 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-envoy" pod "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" (UID: "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.556592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.556532 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-certs" pod "39dd14bb-4cf4-4016-b2b1-59519d96fda7" (UID: "39dd14bb-4cf4-4016-b2b1-59519d96fda7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.557088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.556709 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-data" pod "13833b5b-ab9b-4a32-811b-5eb07645cfa9" (UID: "13833b5b-ab9b-4a32-811b-5eb07645cfa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.557088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.556725 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-data" pod "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" (UID: "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.557088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.556798 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-podinfo" pod "39dd14bb-4cf4-4016-b2b1-59519d96fda7" (UID: "39dd14bb-4cf4-4016-b2b1-59519d96fda7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:54.557088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.556959 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-certs" pod "13833b5b-ab9b-4a32-811b-5eb07645cfa9" (UID: "13833b5b-ab9b-4a32-811b-5eb07645cfa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.557373 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.557277 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istiod-ca-cert" pod "39dd14bb-4cf4-4016-b2b1-59519d96fda7" (UID: "39dd14bb-4cf4-4016-b2b1-59519d96fda7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:54.558235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.557664 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-kube-api-access-lxs2x" pod "39dd14bb-4cf4-4016-b2b1-59519d96fda7" (UID: "39dd14bb-4cf4-4016-b2b1-59519d96fda7"). InnerVolumeSpecName "kube-api-access-lxs2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:54.558235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.557798 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-kube-api-access-hqfls" pod "13833b5b-ab9b-4a32-811b-5eb07645cfa9" (UID: "13833b5b-ab9b-4a32-811b-5eb07645cfa9"). InnerVolumeSpecName "kube-api-access-hqfls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:54.558235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.557925 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-socket" pod "39dd14bb-4cf4-4016-b2b1-59519d96fda7" (UID: "39dd14bb-4cf4-4016-b2b1-59519d96fda7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.558235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.558053 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-envoy" pod "39dd14bb-4cf4-4016-b2b1-59519d96fda7" (UID: "39dd14bb-4cf4-4016-b2b1-59519d96fda7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.558235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.558120 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-credential-socket" pod "13833b5b-ab9b-4a32-811b-5eb07645cfa9" (UID: "13833b5b-ab9b-4a32-811b-5eb07645cfa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.558549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.558254 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-socket" pod "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" (UID: "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.558549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.558272 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-envoy" pod "13833b5b-ab9b-4a32-811b-5eb07645cfa9" (UID: "13833b5b-ab9b-4a32-811b-5eb07645cfa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.558549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.558415 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-podinfo" pod "13833b5b-ab9b-4a32-811b-5eb07645cfa9" (UID: "13833b5b-ab9b-4a32-811b-5eb07645cfa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:54.558549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.558427 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-credential-socket" pod "39dd14bb-4cf4-4016-b2b1-59519d96fda7" (UID: "39dd14bb-4cf4-4016-b2b1-59519d96fda7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:54.558549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.558487 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istiod-ca-cert" pod "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" (UID: "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:54.558893 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.558873 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-podinfo" pod "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" (UID: "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:54.559743 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.559724 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-token" pod "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" (UID: "dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:54.559820 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.559787 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-token" pod "39dd14bb-4cf4-4016-b2b1-59519d96fda7" (UID: "39dd14bb-4cf4-4016-b2b1-59519d96fda7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:54.655308 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655225 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655308 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655258 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655308 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655268 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655308 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655277 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655308 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655286 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655308 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655297 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqfls\" (UniqueName: \"kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-kube-api-access-hqfls\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655308 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655307 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655308 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655315 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fmd6\" (UniqueName: \"kubernetes.io/projected/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-kube-api-access-8fmd6\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655323 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655332 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655340 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13833b5b-ab9b-4a32-811b-5eb07645cfa9-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655349 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxs2x\" (UniqueName: \"kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-kube-api-access-lxs2x\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655356 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655365 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655372 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655380 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655387 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655395 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655403 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13833b5b-ab9b-4a32-811b-5eb07645cfa9-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655411 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655417 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655426 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39dd14bb-4cf4-4016-b2b1-59519d96fda7-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:54.655663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:54.655433 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39dd14bb-4cf4-4016-b2b1-59519d96fda7-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:55.334745 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.334657 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" Oct 30 16:39:55.335224 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.334653 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7" event={"ID":"39dd14bb-4cf4-4016-b2b1-59519d96fda7","Type":"ContainerDied","Data":"3742a8dae575400aa7984a643edbd9e1aace7b03851cee349eb2ad48bb134438"} Oct 30 16:39:55.335224 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.334831 5838 scope.go:122] "RemoveContainer" containerID="434f9907ef5aedf1f52567f9d1e0335a29ae3bc6582f401c39a98e4f4c0afcc2" Oct 30 16:39:55.336244 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.336219 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" Oct 30 16:39:55.336346 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.336243 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px" event={"ID":"dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1","Type":"ContainerDied","Data":"0dcc7e369a5a9a393add35ff48b9cd5b841de72bfbde0c27f207a30e8488b5da"} Oct 30 16:39:55.337999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.337953 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" Oct 30 16:39:55.338098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.338002 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf" event={"ID":"13833b5b-ab9b-4a32-811b-5eb07645cfa9","Type":"ContainerDied","Data":"b096f40c06100a9bed18f65c3231846bc8b2f56c85fe41e769d91365e6f8352f"} Oct 30 16:39:55.339823 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.339805 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" Oct 30 16:39:55.339929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.339837 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t" event={"ID":"d62ed47a-62cf-46a3-84f4-c06c2a13e22c","Type":"ContainerDied","Data":"d3c8812ad29f56e132461018ac2e29b3b55bb1b624944ce5d416d272b71b5516"} Oct 30 16:39:55.349454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.349435 5838 scope.go:122] "RemoveContainer" containerID="553b8b7b7e7299ffa15795a6ddf1011fc0fd5f06ffc3c92941eea6e6e316ffe6" Oct 30 16:39:55.359712 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.359690 5838 scope.go:122] "RemoveContainer" containerID="f8e03bf279a58db19495cecf07019a8e99d7c23180af956907f2b935b9429f6f" Oct 30 16:39:55.369662 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.369645 5838 scope.go:122] "RemoveContainer" containerID="c65f96cb2c08540b928d3fedbb077a96413a748c15e5ce1e1ca73c3fb9292794" Oct 30 16:39:55.398344 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.398317 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf"] Oct 30 16:39:55.409466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.409430 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-5htnf"] Oct 30 16:39:55.437354 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.437324 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t"] Oct 30 16:39:55.441548 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.441522 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-nfx4t"] Oct 30 16:39:55.473194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.473163 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px"] Oct 30 16:39:55.479736 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.479713 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-pd5px"] Oct 30 16:39:55.512118 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.512088 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7"] Oct 30 16:39:55.516620 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:55.516595 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ccf5c9-openshift-default-7bdb68f885-jplq7"] Oct 30 16:39:56.394803 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:56.394767 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" path="/var/lib/kubelet/pods/13833b5b-ab9b-4a32-811b-5eb07645cfa9/volumes" Oct 30 16:39:56.395302 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:56.395285 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39dd14bb-4cf4-4016-b2b1-59519d96fda7" path="/var/lib/kubelet/pods/39dd14bb-4cf4-4016-b2b1-59519d96fda7/volumes" Oct 30 16:39:56.395744 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:56.395727 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" path="/var/lib/kubelet/pods/d62ed47a-62cf-46a3-84f4-c06c2a13e22c/volumes" Oct 30 16:39:56.396549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:56.396530 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" path="/var/lib/kubelet/pods/dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1/volumes" Oct 30 16:39:56.710649 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:56.710572 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:39:56.710649 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:56.710633 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" podUID="63ad2f35-f82b-43d6-b068-163cc3192b26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:39:56.881444 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:56.881403 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" podUID="0795bf8a-e9c9-440f-97a1-d2fdaa47f97f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:39:57.459935 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.459902 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:39:57.460278 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.459990 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" podUID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:39:57.686329 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.686300 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" Oct 30 16:39:57.731544 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.731511 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" Oct 30 16:39:57.789293 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.789257 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-kube-api-access-tw5hc\" (UniqueName: \"kubernetes.io/projected/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-kube-api-access-tw5hc\") pod \"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f\" (UID: \"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f\") " Oct 30 16:39:57.789477 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.789320 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-kube-api-access-w8pgr\" (UniqueName: \"kubernetes.io/projected/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-kube-api-access-w8pgr\") pod \"7203fda6-f9e6-4d6f-96da-0514b72e7ca7\" (UID: \"7203fda6-f9e6-4d6f-96da-0514b72e7ca7\") " Oct 30 16:39:57.789477 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.789460 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-plugins-config-volume\") pod \"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f\" (UID: \"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f\") " Oct 30 16:39:57.789566 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.789503 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-plugins-config-volume\") pod \"7203fda6-f9e6-4d6f-96da-0514b72e7ca7\" (UID: \"7203fda6-f9e6-4d6f-96da-0514b72e7ca7\") " Oct 30 16:39:57.789943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.789911 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-plugins-config-volume" pod "0795bf8a-e9c9-440f-97a1-d2fdaa47f97f" (UID: "0795bf8a-e9c9-440f-97a1-d2fdaa47f97f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:57.790187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.790165 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-plugins-config-volume" pod "7203fda6-f9e6-4d6f-96da-0514b72e7ca7" (UID: "7203fda6-f9e6-4d6f-96da-0514b72e7ca7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:57.791388 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.791368 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-kube-api-access-w8pgr" pod "7203fda6-f9e6-4d6f-96da-0514b72e7ca7" (UID: "7203fda6-f9e6-4d6f-96da-0514b72e7ca7"). InnerVolumeSpecName "kube-api-access-w8pgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:57.791451 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.791435 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-kube-api-access-tw5hc" pod "0795bf8a-e9c9-440f-97a1-d2fdaa47f97f" (UID: "0795bf8a-e9c9-440f-97a1-d2fdaa47f97f"). InnerVolumeSpecName "kube-api-access-tw5hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:57.890994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.890926 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:57.890994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.890964 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:57.891209 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.891012 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw5hc\" (UniqueName: \"kubernetes.io/projected/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f-kube-api-access-tw5hc\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:57.891209 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.891027 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8pgr\" (UniqueName: \"kubernetes.io/projected/7203fda6-f9e6-4d6f-96da-0514b72e7ca7-kube-api-access-w8pgr\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:57.902792 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.902716 5838 patch_prober.go:29] interesting pod/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:39:57.902916 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:57.902781 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:39:58.379404 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.379367 5838 generic.go:356] "Generic (PLEG): container finished" podID="0795bf8a-e9c9-440f-97a1-d2fdaa47f97f" containerID="1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163" exitCode=0 Oct 30 16:39:58.379616 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.379415 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" Oct 30 16:39:58.379616 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.379450 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" event={"ID":"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f","Type":"ContainerDied","Data":"1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163"} Oct 30 16:39:58.379616 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.379545 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4" event={"ID":"0795bf8a-e9c9-440f-97a1-d2fdaa47f97f","Type":"ContainerDied","Data":"3a781507daaba980586ed0ba63b3be9a073ad02f9e8ccf8dcc530ee0f6cbd0be"} Oct 30 16:39:58.379616 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.379576 5838 scope.go:122] "RemoveContainer" containerID="1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163" Oct 30 16:39:58.387516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.387479 5838 generic.go:356] "Generic (PLEG): container finished" podID="7203fda6-f9e6-4d6f-96da-0514b72e7ca7" containerID="e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba" exitCode=0 Oct 30 16:39:58.387682 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.387567 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" Oct 30 16:39:58.387682 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.387561 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" event={"ID":"7203fda6-f9e6-4d6f-96da-0514b72e7ca7","Type":"ContainerDied","Data":"e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba"} Oct 30 16:39:58.387801 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.387709 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p" event={"ID":"7203fda6-f9e6-4d6f-96da-0514b72e7ca7","Type":"ContainerDied","Data":"1787f4f0e8e1599ee28fb0c17c8bdb102b06f3c658d6386ddcd8943b7bd621f8"} Oct 30 16:39:58.398528 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.398503 5838 scope.go:122] "RemoveContainer" containerID="1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163" Oct 30 16:39:58.400308 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:58.400276 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163\": container with ID starting with 1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163 not found: ID does not exist" containerID="1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163" Oct 30 16:39:58.400416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.400319 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163"} err="failed to get container status \"1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163\": rpc error: code = NotFound desc = could not find container \"1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163\": container with ID starting with 1073793a2484bcac74b8633f7629b5875b360ba2655847e03d40b0aaea0f0163 not found: ID does not exist" Oct 30 16:39:58.400416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.400343 5838 scope.go:122] "RemoveContainer" containerID="e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba" Oct 30 16:39:58.442721 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.442692 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4"] Oct 30 16:39:58.453596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.453569 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b55521-574455d8dc-96mc4"] Oct 30 16:39:58.659791 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.659435 5838 scope.go:122] "RemoveContainer" containerID="e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba" Oct 30 16:39:58.661266 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:58.661043 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba\": container with ID starting with e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba not found: ID does not exist" containerID="e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba" Oct 30 16:39:58.661266 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.661111 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba"} err="failed to get container status \"e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba\": rpc error: code = NotFound desc = could not find container \"e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba\": container with ID starting with e4d3ad79ba33d7d170b12f86b31356dbee33dcf4c6aa2a9bebe17582d1f11dba not found: ID does not exist" Oct 30 16:39:58.698084 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.698001 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p"] Oct 30 16:39:58.707478 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.707379 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b55521-65c57dbf97-2wv7p"] Oct 30 16:39:58.722617 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.722258 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:39:58.783736 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.782782 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:39:58.809342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.808101 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-socket\") pod \"e8afe9b1-9212-4f51-874a-cb2520b26eae\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " Oct 30 16:39:58.809342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.808176 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-credential-socket\") pod \"e8afe9b1-9212-4f51-874a-cb2520b26eae\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " Oct 30 16:39:58.809342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.808217 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-kube-api-access-n24vt\" (UniqueName: \"kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-kube-api-access-n24vt\") pod \"e8afe9b1-9212-4f51-874a-cb2520b26eae\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " Oct 30 16:39:58.809342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.808299 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8afe9b1-9212-4f51-874a-cb2520b26eae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8afe9b1-9212-4f51-874a-cb2520b26eae-istiod-ca-cert\") pod \"e8afe9b1-9212-4f51-874a-cb2520b26eae\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " Oct 30 16:39:58.809342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.808359 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-data\") pod \"e8afe9b1-9212-4f51-874a-cb2520b26eae\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " Oct 30 16:39:58.809342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.808462 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-certs\") pod \"e8afe9b1-9212-4f51-874a-cb2520b26eae\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " Oct 30 16:39:58.809342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.808497 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-envoy\") pod \"e8afe9b1-9212-4f51-874a-cb2520b26eae\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " Oct 30 16:39:58.809342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.808527 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-podinfo\") pod \"e8afe9b1-9212-4f51-874a-cb2520b26eae\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " Oct 30 16:39:58.809342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.808564 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-token\" (UniqueName: \"kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-token\") pod \"e8afe9b1-9212-4f51-874a-cb2520b26eae\" (UID: \"e8afe9b1-9212-4f51-874a-cb2520b26eae\") " Oct 30 16:39:58.811098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.810523 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-certs" pod "e8afe9b1-9212-4f51-874a-cb2520b26eae" (UID: "e8afe9b1-9212-4f51-874a-cb2520b26eae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:58.813188 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.812730 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-credential-socket" pod "e8afe9b1-9212-4f51-874a-cb2520b26eae" (UID: "e8afe9b1-9212-4f51-874a-cb2520b26eae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:58.813188 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.812948 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-socket" pod "e8afe9b1-9212-4f51-874a-cb2520b26eae" (UID: "e8afe9b1-9212-4f51-874a-cb2520b26eae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:58.813714 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.813415 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-data" pod "e8afe9b1-9212-4f51-874a-cb2520b26eae" (UID: "e8afe9b1-9212-4f51-874a-cb2520b26eae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:58.814026 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.813983 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-token" pod "e8afe9b1-9212-4f51-874a-cb2520b26eae" (UID: "e8afe9b1-9212-4f51-874a-cb2520b26eae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:58.815101 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.815079 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8afe9b1-9212-4f51-874a-cb2520b26eae-istiod-ca-cert" pod "e8afe9b1-9212-4f51-874a-cb2520b26eae" (UID: "e8afe9b1-9212-4f51-874a-cb2520b26eae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:58.837122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.837082 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-envoy" pod "e8afe9b1-9212-4f51-874a-cb2520b26eae" (UID: "e8afe9b1-9212-4f51-874a-cb2520b26eae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:58.851155 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.851102 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-podinfo" pod "e8afe9b1-9212-4f51-874a-cb2520b26eae" (UID: "e8afe9b1-9212-4f51-874a-cb2520b26eae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:58.851339 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.851232 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-kube-api-access-n24vt" pod "e8afe9b1-9212-4f51-874a-cb2520b26eae" (UID: "e8afe9b1-9212-4f51-874a-cb2520b26eae"). InnerVolumeSpecName "kube-api-access-n24vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:58.910234 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.910199 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-certs\") pod \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " Oct 30 16:39:58.910719 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.910696 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-podinfo\") pod \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " Oct 30 16:39:58.911138 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.911115 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istiod-ca-cert\") pod \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " Oct 30 16:39:58.911744 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.910635 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-certs" pod "1c251b61-eafd-43e1-b6b6-a91f4ba965ff" (UID: "1c251b61-eafd-43e1-b6b6-a91f4ba965ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:58.911890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.911865 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istiod-ca-cert" pod "1c251b61-eafd-43e1-b6b6-a91f4ba965ff" (UID: "1c251b61-eafd-43e1-b6b6-a91f4ba965ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:58.912171 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.912151 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-envoy\") pod \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " Oct 30 16:39:58.915681 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.915661 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-data\") pod \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " Oct 30 16:39:58.916724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.916409 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-kube-api-access-nkjmr\" (UniqueName: \"kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-kube-api-access-nkjmr\") pod \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " Oct 30 16:39:58.916919 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.916901 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-credential-socket\") pod \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " Oct 30 16:39:58.917445 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.915434 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-envoy" pod "1c251b61-eafd-43e1-b6b6-a91f4ba965ff" (UID: "1c251b61-eafd-43e1-b6b6-a91f4ba965ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:58.917445 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.915581 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-podinfo" pod "1c251b61-eafd-43e1-b6b6-a91f4ba965ff" (UID: "1c251b61-eafd-43e1-b6b6-a91f4ba965ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:58.917445 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.916357 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-data" pod "1c251b61-eafd-43e1-b6b6-a91f4ba965ff" (UID: "1c251b61-eafd-43e1-b6b6-a91f4ba965ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:58.917445 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.917256 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-credential-socket" pod "1c251b61-eafd-43e1-b6b6-a91f4ba965ff" (UID: "1c251b61-eafd-43e1-b6b6-a91f4ba965ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:58.920172 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.920150 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-kube-api-access-nkjmr" pod "1c251b61-eafd-43e1-b6b6-a91f4ba965ff" (UID: "1c251b61-eafd-43e1-b6b6-a91f4ba965ff"). InnerVolumeSpecName "kube-api-access-nkjmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:58.920337 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.920320 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-socket\") pod \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " Oct 30 16:39:58.920658 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.920643 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-token\" (UniqueName: \"kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-token\") pod \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\" (UID: \"1c251b61-eafd-43e1-b6b6-a91f4ba965ff\") " Oct 30 16:39:58.921705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.921683 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.921889 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.920901 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.920603 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-socket" pod "1c251b61-eafd-43e1-b6b6-a91f4ba965ff" (UID: "1c251b61-eafd-43e1-b6b6-a91f4ba965ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922011 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922177 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922194 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922210 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922223 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922236 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922250 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922265 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922281 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8afe9b1-9212-4f51-874a-cb2520b26eae-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922296 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922310 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n24vt\" (UniqueName: \"kubernetes.io/projected/e8afe9b1-9212-4f51-874a-cb2520b26eae-kube-api-access-n24vt\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922325 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922339 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkjmr\" (UniqueName: \"kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-kube-api-access-nkjmr\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.922400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.922353 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8afe9b1-9212-4f51-874a-cb2520b26eae-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.923837 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.923808 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:58.924058 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.924026 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-token" pod "1c251b61-eafd-43e1-b6b6-a91f4ba965ff" (UID: "1c251b61-eafd-43e1-b6b6-a91f4ba965ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:58.924239 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:58.924223 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:39:59.032846 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.032809 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-podinfo\") pod \"bad79af6-22c9-4997-a3cf-2090794ce3e9\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " Oct 30 16:39:59.033065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.032859 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-certs\") pod \"63ad2f35-f82b-43d6-b068-163cc3192b26\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " Oct 30 16:39:59.033065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.032888 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-kube-api-access-pchxl\" (UniqueName: \"kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-kube-api-access-pchxl\") pod \"bad79af6-22c9-4997-a3cf-2090794ce3e9\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " Oct 30 16:39:59.033065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.032924 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-envoy\") pod \"bad79af6-22c9-4997-a3cf-2090794ce3e9\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " Oct 30 16:39:59.033065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.032951 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-data\") pod \"63ad2f35-f82b-43d6-b068-163cc3192b26\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " Oct 30 16:39:59.033065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.033011 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-credential-socket\") pod \"bad79af6-22c9-4997-a3cf-2090794ce3e9\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " Oct 30 16:39:59.033065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.033047 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-socket\") pod \"bad79af6-22c9-4997-a3cf-2090794ce3e9\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " Oct 30 16:39:59.033399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.033110 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-kube-api-access-mwjks\" (UniqueName: \"kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-kube-api-access-mwjks\") pod \"63ad2f35-f82b-43d6-b068-163cc3192b26\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " Oct 30 16:39:59.033399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.033152 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-token\" (UniqueName: \"kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-token\") pod \"bad79af6-22c9-4997-a3cf-2090794ce3e9\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " Oct 30 16:39:59.033399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.033206 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-envoy\") pod \"63ad2f35-f82b-43d6-b068-163cc3192b26\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " Oct 30 16:39:59.033399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.033258 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ad2f35-f82b-43d6-b068-163cc3192b26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ad2f35-f82b-43d6-b068-163cc3192b26-istiod-ca-cert\") pod \"63ad2f35-f82b-43d6-b068-163cc3192b26\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " Oct 30 16:39:59.033399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.033291 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-podinfo\") pod \"63ad2f35-f82b-43d6-b068-163cc3192b26\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " Oct 30 16:39:59.033399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.033340 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-credential-socket\") pod \"63ad2f35-f82b-43d6-b068-163cc3192b26\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " Oct 30 16:39:59.033399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.033370 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad79af6-22c9-4997-a3cf-2090794ce3e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad79af6-22c9-4997-a3cf-2090794ce3e9-istiod-ca-cert\") pod \"bad79af6-22c9-4997-a3cf-2090794ce3e9\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " Oct 30 16:39:59.033747 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.033417 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-socket\") pod \"63ad2f35-f82b-43d6-b068-163cc3192b26\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " Oct 30 16:39:59.033747 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.033678 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-socket" pod "63ad2f35-f82b-43d6-b068-163cc3192b26" (UID: "63ad2f35-f82b-43d6-b068-163cc3192b26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:59.034162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.034136 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad79af6-22c9-4997-a3cf-2090794ce3e9-istiod-ca-cert" pod "bad79af6-22c9-4997-a3cf-2090794ce3e9" (UID: "bad79af6-22c9-4997-a3cf-2090794ce3e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:59.034162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.034141 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-credential-socket" pod "63ad2f35-f82b-43d6-b068-163cc3192b26" (UID: "63ad2f35-f82b-43d6-b068-163cc3192b26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:59.034333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.034208 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-certs\") pod \"bad79af6-22c9-4997-a3cf-2090794ce3e9\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " Oct 30 16:39:59.034333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.034243 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-token\" (UniqueName: \"kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-token\") pod \"63ad2f35-f82b-43d6-b068-163cc3192b26\" (UID: \"63ad2f35-f82b-43d6-b068-163cc3192b26\") " Oct 30 16:39:59.034333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.034273 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-data\") pod \"bad79af6-22c9-4997-a3cf-2090794ce3e9\" (UID: \"bad79af6-22c9-4997-a3cf-2090794ce3e9\") " Oct 30 16:39:59.034517 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.034494 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ad2f35-f82b-43d6-b068-163cc3192b26-istiod-ca-cert" pod "63ad2f35-f82b-43d6-b068-163cc3192b26" (UID: "63ad2f35-f82b-43d6-b068-163cc3192b26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:59.034707 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.034686 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-data" pod "bad79af6-22c9-4997-a3cf-2090794ce3e9" (UID: "bad79af6-22c9-4997-a3cf-2090794ce3e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:59.034838 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.034797 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.034940 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.034926 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c251b61-eafd-43e1-b6b6-a91f4ba965ff-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.035065 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.035049 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ad2f35-f82b-43d6-b068-163cc3192b26-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.035349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.035140 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-certs" pod "63ad2f35-f82b-43d6-b068-163cc3192b26" (UID: "63ad2f35-f82b-43d6-b068-163cc3192b26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:59.035349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.035161 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.035349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.035179 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bad79af6-22c9-4997-a3cf-2090794ce3e9-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.035349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.035193 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.035820 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.035636 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-certs" pod "bad79af6-22c9-4997-a3cf-2090794ce3e9" (UID: "bad79af6-22c9-4997-a3cf-2090794ce3e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:59.036015 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.035952 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-data" pod "63ad2f35-f82b-43d6-b068-163cc3192b26" (UID: "63ad2f35-f82b-43d6-b068-163cc3192b26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:59.037213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.036963 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-podinfo" pod "bad79af6-22c9-4997-a3cf-2090794ce3e9" (UID: "bad79af6-22c9-4997-a3cf-2090794ce3e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:59.037520 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.037345 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-credential-socket" pod "bad79af6-22c9-4997-a3cf-2090794ce3e9" (UID: "bad79af6-22c9-4997-a3cf-2090794ce3e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:59.037677 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.037637 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-socket" pod "bad79af6-22c9-4997-a3cf-2090794ce3e9" (UID: "bad79af6-22c9-4997-a3cf-2090794ce3e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:59.038354 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.038324 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-envoy" pod "bad79af6-22c9-4997-a3cf-2090794ce3e9" (UID: "bad79af6-22c9-4997-a3cf-2090794ce3e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:59.039164 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.039136 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-podinfo" pod "63ad2f35-f82b-43d6-b068-163cc3192b26" (UID: "63ad2f35-f82b-43d6-b068-163cc3192b26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:39:59.039164 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.039141 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-kube-api-access-pchxl" pod "bad79af6-22c9-4997-a3cf-2090794ce3e9" (UID: "bad79af6-22c9-4997-a3cf-2090794ce3e9"). InnerVolumeSpecName "kube-api-access-pchxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:59.039311 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.039266 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-kube-api-access-mwjks" pod "63ad2f35-f82b-43d6-b068-163cc3192b26" (UID: "63ad2f35-f82b-43d6-b068-163cc3192b26"). InnerVolumeSpecName "kube-api-access-mwjks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:59.040076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.040050 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-token" pod "bad79af6-22c9-4997-a3cf-2090794ce3e9" (UID: "bad79af6-22c9-4997-a3cf-2090794ce3e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:59.040936 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.040909 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-envoy" pod "63ad2f35-f82b-43d6-b068-163cc3192b26" (UID: "63ad2f35-f82b-43d6-b068-163cc3192b26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:39:59.041098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.041080 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-token" pod "63ad2f35-f82b-43d6-b068-163cc3192b26" (UID: "63ad2f35-f82b-43d6-b068-163cc3192b26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:59.136798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136743 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.136798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136790 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.136798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136805 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.137253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136821 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.137253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136835 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.137253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136848 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.137253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136863 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.137253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136876 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.137253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136890 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pchxl\" (UniqueName: \"kubernetes.io/projected/bad79af6-22c9-4997-a3cf-2090794ce3e9-kube-api-access-pchxl\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.137253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136906 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.137253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136919 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ad2f35-f82b-43d6-b068-163cc3192b26-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.137253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136932 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.137253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136945 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bad79af6-22c9-4997-a3cf-2090794ce3e9-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.137253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.136961 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwjks\" (UniqueName: \"kubernetes.io/projected/63ad2f35-f82b-43d6-b068-163cc3192b26-kube-api-access-mwjks\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:39:59.431187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.418875 5838 generic.go:356] "Generic (PLEG): container finished" podID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" containerID="506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192" exitCode=0 Oct 30 16:39:59.431187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.419061 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" event={"ID":"1c251b61-eafd-43e1-b6b6-a91f4ba965ff","Type":"ContainerDied","Data":"506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192"} Oct 30 16:39:59.431187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.419123 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" event={"ID":"1c251b61-eafd-43e1-b6b6-a91f4ba965ff","Type":"ContainerDied","Data":"0a8054e196490ff0da9246b5f5b697de182d7eea83ee7637182a3b54fcd74126"} Oct 30 16:39:59.431187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.419154 5838 scope.go:122] "RemoveContainer" containerID="506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192" Oct 30 16:39:59.431187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.419286 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj" Oct 30 16:39:59.449167 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.443583 5838 generic.go:356] "Generic (PLEG): container finished" podID="63ad2f35-f82b-43d6-b068-163cc3192b26" containerID="b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d" exitCode=0 Oct 30 16:39:59.449167 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.443734 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" event={"ID":"63ad2f35-f82b-43d6-b068-163cc3192b26","Type":"ContainerDied","Data":"b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d"} Oct 30 16:39:59.449167 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.443790 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" event={"ID":"63ad2f35-f82b-43d6-b068-163cc3192b26","Type":"ContainerDied","Data":"0671a403bbba6cde40913224e1290d1ebf7af05407ce6b2a6b0b686a92403355"} Oct 30 16:39:59.449167 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.443879 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d" Oct 30 16:39:59.465162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.452153 5838 generic.go:356] "Generic (PLEG): container finished" podID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerID="49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e" exitCode=0 Oct 30 16:39:59.465162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.452258 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" event={"ID":"e8afe9b1-9212-4f51-874a-cb2520b26eae","Type":"ContainerDied","Data":"49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e"} Oct 30 16:39:59.465162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.452307 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" event={"ID":"e8afe9b1-9212-4f51-874a-cb2520b26eae","Type":"ContainerDied","Data":"f9ae516c29c478457b5c93ab66e04ce16c208bd6dcb5c98f42615625edfbdf7e"} Oct 30 16:39:59.465162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.452393 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg" Oct 30 16:39:59.465162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.460788 5838 generic.go:356] "Generic (PLEG): container finished" podID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerID="cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6" exitCode=0 Oct 30 16:39:59.465162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.460839 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" event={"ID":"bad79af6-22c9-4997-a3cf-2090794ce3e9","Type":"ContainerDied","Data":"cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6"} Oct 30 16:39:59.465162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.460882 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" event={"ID":"bad79af6-22c9-4997-a3cf-2090794ce3e9","Type":"ContainerDied","Data":"533a0f717db6fe519b3aa46297fdcee38cca70059d3b65f567cc98072054e0f0"} Oct 30 16:39:59.465162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.460960 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd" Oct 30 16:39:59.503933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.500778 5838 scope.go:122] "RemoveContainer" containerID="506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192" Oct 30 16:39:59.503933 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:59.502933 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192\": container with ID starting with 506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192 not found: ID does not exist" containerID="506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192" Oct 30 16:39:59.503933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.502998 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192"} err="failed to get container status \"506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192\": rpc error: code = NotFound desc = could not find container \"506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192\": container with ID starting with 506671e28fa5f5553687aaa1026835df52011252ac4c2ac62d1b5d9bbffb1192 not found: ID does not exist" Oct 30 16:39:59.503933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.503022 5838 scope.go:122] "RemoveContainer" containerID="b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d" Oct 30 16:39:59.574004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.566843 5838 scope.go:122] "RemoveContainer" containerID="b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d" Oct 30 16:39:59.574004 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:59.567326 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d\": container with ID starting with b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d not found: ID does not exist" containerID="b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d" Oct 30 16:39:59.574004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.567377 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d"} err="failed to get container status \"b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d\": rpc error: code = NotFound desc = could not find container \"b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d\": container with ID starting with b8c60935dffc42cc3ea1aeb9ea2fe73d9571dd3e1cc382fcb6be0cde18ba908d not found: ID does not exist" Oct 30 16:39:59.574004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.567402 5838 scope.go:122] "RemoveContainer" containerID="49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e" Oct 30 16:39:59.590251 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.590212 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd"] Oct 30 16:39:59.601056 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.601026 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-j6ghd"] Oct 30 16:39:59.605235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.605217 5838 scope.go:122] "RemoveContainer" containerID="49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e" Oct 30 16:39:59.605706 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:59.605681 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e\": container with ID starting with 49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e not found: ID does not exist" containerID="49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e" Oct 30 16:39:59.605798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.605714 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e"} err="failed to get container status \"49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e\": rpc error: code = NotFound desc = could not find container \"49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e\": container with ID starting with 49c86c1a65317bbc231b57a83cc6a3431d0953766d12fd9bec5869000732c58e not found: ID does not exist" Oct 30 16:39:59.605798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.605732 5838 scope.go:122] "RemoveContainer" containerID="cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6" Oct 30 16:39:59.607523 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.607420 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj"] Oct 30 16:39:59.611552 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.611530 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-qpvhj"] Oct 30 16:39:59.626576 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.626553 5838 scope.go:122] "RemoveContainer" containerID="d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe" Oct 30 16:39:59.692859 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.692768 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg"] Oct 30 16:39:59.705408 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.705377 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-wtpxg"] Oct 30 16:39:59.712365 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.711618 5838 scope.go:122] "RemoveContainer" containerID="cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6" Oct 30 16:39:59.712365 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:59.712323 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6\": container with ID starting with cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6 not found: ID does not exist" containerID="cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6" Oct 30 16:39:59.712532 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.712358 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6"} err="failed to get container status \"cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6\": rpc error: code = NotFound desc = could not find container \"cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6\": container with ID starting with cb0bd2ef57cfea59d79a646257f0af3b3a3276df6d33088d1d6a9104249f7ea6 not found: ID does not exist" Oct 30 16:39:59.712532 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.712383 5838 scope.go:122] "RemoveContainer" containerID="d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe" Oct 30 16:39:59.712804 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:39:59.712780 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe\": container with ID starting with d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe not found: ID does not exist" containerID="d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe" Oct 30 16:39:59.712870 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.712816 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe"} err="failed to get container status \"d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe\": rpc error: code = NotFound desc = could not find container \"d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe\": container with ID starting with d2aeeda5273eace40c41bff7600a887814ab15029875c2a1f79ffdfc54e681fe not found: ID does not exist" Oct 30 16:39:59.713122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.713102 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d"] Oct 30 16:39:59.752000 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:39:59.739959 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b55521-openshift-default-79969cd999-llm6d"] Oct 30 16:40:00.397640 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:00.397609 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0795bf8a-e9c9-440f-97a1-d2fdaa47f97f" path="/var/lib/kubelet/pods/0795bf8a-e9c9-440f-97a1-d2fdaa47f97f/volumes" Oct 30 16:40:00.398111 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:00.398097 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" path="/var/lib/kubelet/pods/1c251b61-eafd-43e1-b6b6-a91f4ba965ff/volumes" Oct 30 16:40:00.398490 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:00.398478 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ad2f35-f82b-43d6-b068-163cc3192b26" path="/var/lib/kubelet/pods/63ad2f35-f82b-43d6-b068-163cc3192b26/volumes" Oct 30 16:40:00.399135 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:00.399120 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7203fda6-f9e6-4d6f-96da-0514b72e7ca7" path="/var/lib/kubelet/pods/7203fda6-f9e6-4d6f-96da-0514b72e7ca7/volumes" Oct 30 16:40:00.399527 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:00.399514 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" path="/var/lib/kubelet/pods/bad79af6-22c9-4997-a3cf-2090794ce3e9/volumes" Oct 30 16:40:00.399984 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:00.399953 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" path="/var/lib/kubelet/pods/e8afe9b1-9212-4f51-874a-cb2520b26eae/volumes" Oct 30 16:40:01.362417 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362384 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85bb7dd889-q9jfl"] Oct 30 16:40:01.362808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362770 5838 cpu_manager.go:414] "containerMap: removing container" podUID="dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" containerName="istio-proxy" Oct 30 16:40:01.362808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362781 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" containerName="istio-proxy" Oct 30 16:40:01.362808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362788 5838 cpu_manager.go:414] "containerMap: removing container" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerName="istio-proxy" Oct 30 16:40:01.362808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362793 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerName="istio-proxy" Oct 30 16:40:01.362808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362805 5838 cpu_manager.go:414] "containerMap: removing container" podUID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" containerName="istio-proxy" Oct 30 16:40:01.362808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362809 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362818 5838 cpu_manager.go:414] "containerMap: removing container" podUID="9dc48445-4345-4a22-bc03-6c2108da11b6" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362823 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc48445-4345-4a22-bc03-6c2108da11b6" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362832 5838 cpu_manager.go:414] "containerMap: removing container" podUID="0795bf8a-e9c9-440f-97a1-d2fdaa47f97f" containerName="payload-pre-processing" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362837 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0795bf8a-e9c9-440f-97a1-d2fdaa47f97f" containerName="payload-pre-processing" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362846 5838 cpu_manager.go:414] "containerMap: removing container" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362850 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362859 5838 cpu_manager.go:414] "containerMap: removing container" podUID="7203fda6-f9e6-4d6f-96da-0514b72e7ca7" containerName="payload-processing" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362864 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7203fda6-f9e6-4d6f-96da-0514b72e7ca7" containerName="payload-processing" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362871 5838 cpu_manager.go:414] "containerMap: removing container" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362875 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362884 5838 cpu_manager.go:414] "containerMap: removing container" podUID="a4d91ca0-9721-4ff9-8831-18a73e08b207" containerName="payload-processing" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362889 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d91ca0-9721-4ff9-8831-18a73e08b207" containerName="payload-processing" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362896 5838 cpu_manager.go:414] "containerMap: removing container" podUID="39dd14bb-4cf4-4016-b2b1-59519d96fda7" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362900 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39dd14bb-4cf4-4016-b2b1-59519d96fda7" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362907 5838 cpu_manager.go:414] "containerMap: removing container" podUID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362912 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362921 5838 cpu_manager.go:414] "containerMap: removing container" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362926 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362934 5838 cpu_manager.go:414] "containerMap: removing container" podUID="9ff126c2-5d26-47d2-8440-8d81421441de" containerName="revoke-keys" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362938 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff126c2-5d26-47d2-8440-8d81421441de" containerName="revoke-keys" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362946 5838 cpu_manager.go:414] "containerMap: removing container" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362950 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362957 5838 cpu_manager.go:414] "containerMap: removing container" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362961 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362979 5838 cpu_manager.go:414] "containerMap: removing container" podUID="fde47769-223e-49c6-b4e9-b5c9295d8d85" containerName="payload-pre-processing" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362984 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde47769-223e-49c6-b4e9-b5c9295d8d85" containerName="payload-pre-processing" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362994 5838 cpu_manager.go:414] "containerMap: removing container" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.362998 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363007 5838 cpu_manager.go:414] "containerMap: removing container" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363011 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363017 5838 cpu_manager.go:414] "containerMap: removing container" podUID="63ad2f35-f82b-43d6-b068-163cc3192b26" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363022 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ad2f35-f82b-43d6-b068-163cc3192b26" containerName="istio-proxy" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363030 5838 cpu_manager.go:414] "containerMap: removing container" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363034 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" Oct 30 16:40:01.363024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363040 5838 cpu_manager.go:414] "containerMap: removing container" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363045 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363052 5838 cpu_manager.go:414] "containerMap: removing container" podUID="7d7a25b6-d5b4-470f-b5a6-66b57f2545d3" containerName="revoke-keys" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363056 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7a25b6-d5b4-470f-b5a6-66b57f2545d3" containerName="revoke-keys" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363062 5838 cpu_manager.go:414] "containerMap: removing container" podUID="e20e8844-c4fd-40fb-884e-5c53aa339890" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363066 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20e8844-c4fd-40fb-884e-5c53aa339890" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363074 5838 cpu_manager.go:414] "containerMap: removing container" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363078 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363085 5838 cpu_manager.go:414] "containerMap: removing container" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363088 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363144 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363150 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62ed47a-62cf-46a3-84f4-c06c2a13e22c" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363157 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbf167c-2d5f-4cf5-a35f-d3d33ce303d1" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363163 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8afe9b1-9212-4f51-874a-cb2520b26eae" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363170 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="13833b5b-ab9b-4a32-811b-5eb07645cfa9" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363175 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc48445-4345-4a22-bc03-6c2108da11b6" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363180 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="0795bf8a-e9c9-440f-97a1-d2fdaa47f97f" containerName="payload-pre-processing" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363186 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a55a46-09fd-47f5-b9dd-d725aab9bfb2" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363191 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="7203fda6-f9e6-4d6f-96da-0514b72e7ca7" containerName="payload-processing" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363196 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363201 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d91ca0-9721-4ff9-8831-18a73e08b207" containerName="payload-processing" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363206 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="39dd14bb-4cf4-4016-b2b1-59519d96fda7" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363211 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c251b61-eafd-43e1-b6b6-a91f4ba965ff" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363216 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="340ac92b-1be2-4337-a547-c0c0d239ca96" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363221 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ff126c2-5d26-47d2-8440-8d81421441de" containerName="revoke-keys" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363225 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363229 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad79af6-22c9-4997-a3cf-2090794ce3e9" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363234 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde47769-223e-49c6-b4e9-b5c9295d8d85" containerName="payload-pre-processing" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363239 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa5b1376-67d3-4de7-abce-93884b4eb2ba" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363245 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6f8643-fe14-47eb-81f1-bd89bd12bdbe" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363250 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ad2f35-f82b-43d6-b068-163cc3192b26" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363255 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e579322a-2a82-46bd-8959-e75ee51bb57d" containerName="main" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363259 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="439bafeb-d93c-46f1-a33f-1a1ffcd1d776" containerName="istio-proxy" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363264 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7a25b6-d5b4-470f-b5a6-66b57f2545d3" containerName="revoke-keys" Oct 30 16:40:01.363993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.363271 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20e8844-c4fd-40fb-884e-5c53aa339890" containerName="istio-proxy" Oct 30 16:40:01.368538 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.368514 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:01.395791 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.395749 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 16:40:01.395791 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.395749 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 16:40:01.396124 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.396097 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lvg8k\"" Oct 30 16:40:01.396308 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.396289 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 16:40:01.400265 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.400239 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85bb7dd889-q9jfl"] Oct 30 16:40:01.497061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.497031 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-metrics-certs\") pod \"maas-controller-85bb7dd889-q9jfl\" (UID: \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\") " pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:01.497206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.497069 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-webhook-cert\") pod \"maas-controller-85bb7dd889-q9jfl\" (UID: \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\") " pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:01.497206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.497094 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvfr\" (UniqueName: \"kubernetes.io/projected/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-kube-api-access-qdvfr\") pod \"maas-controller-85bb7dd889-q9jfl\" (UID: \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\") " pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:01.597729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.597682 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-metrics-certs\") pod \"maas-controller-85bb7dd889-q9jfl\" (UID: \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\") " pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:01.597729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.597733 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-webhook-cert\") pod \"maas-controller-85bb7dd889-q9jfl\" (UID: \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\") " pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:01.597924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.597822 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvfr\" (UniqueName: \"kubernetes.io/projected/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-kube-api-access-qdvfr\") pod \"maas-controller-85bb7dd889-q9jfl\" (UID: \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\") " pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:01.600332 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.600309 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-webhook-cert\") pod \"maas-controller-85bb7dd889-q9jfl\" (UID: \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\") " pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:01.600439 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.600418 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-metrics-certs\") pod \"maas-controller-85bb7dd889-q9jfl\" (UID: \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\") " pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:01.620574 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.620507 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvfr\" (UniqueName: \"kubernetes.io/projected/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-kube-api-access-qdvfr\") pod \"maas-controller-85bb7dd889-q9jfl\" (UID: \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\") " pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:01.682590 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.682555 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:01.796496 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:40:01.796460 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:40:01.990957 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.990931 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85bb7dd889-q9jfl"] Oct 30 16:40:01.993114 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:40:01.993085 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50d8645_b4aa_43f3_a079_f6c61a2bd38b.slice/crio-3be1e02ed8ab32ef03a70e7ae06beca39c1f4a1ea86f35475f12f0cbd42a01ba WatchSource:0}: Error finding container 3be1e02ed8ab32ef03a70e7ae06beca39c1f4a1ea86f35475f12f0cbd42a01ba: Status 404 returned error can't find the container with id 3be1e02ed8ab32ef03a70e7ae06beca39c1f4a1ea86f35475f12f0cbd42a01ba Oct 30 16:40:01.994490 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:01.994471 5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:40:02.510892 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:02.510857 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85bb7dd889-q9jfl" event={"ID":"a50d8645-b4aa-43f3-a079-f6c61a2bd38b","Type":"ContainerStarted","Data":"3be1e02ed8ab32ef03a70e7ae06beca39c1f4a1ea86f35475f12f0cbd42a01ba"} Oct 30 16:40:04.040868 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:04.040768 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j"] Oct 30 16:40:04.050818 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:04.050784 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81d33ab8-v7v9j"] Oct 30 16:40:04.394466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:04.394389 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d35e40-a13d-4e71-ab84-4ee846e50241" path="/var/lib/kubelet/pods/c5d35e40-a13d-4e71-ab84-4ee846e50241/volumes" Oct 30 16:40:04.554102 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:04.554057 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85bb7dd889-q9jfl" event={"ID":"a50d8645-b4aa-43f3-a079-f6c61a2bd38b","Type":"ContainerStarted","Data":"27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf"} Oct 30 16:40:04.554310 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:04.554207 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:04.597371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:04.597308 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-85bb7dd889-q9jfl" podStartSLOduration=1.8040014819999999 podStartE2EDuration="3.597290449s" totalImagesPullingTime="1.793288967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:04.594824235 +0000 UTC m=+1166.343413645" watchObservedRunningTime="2026-10-30 16:40:04.597290449 +0000 UTC m=+1166.345879838" Oct 30 16:40:05.029901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:05.029854 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb"] Oct 30 16:40:05.034470 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:05.034437 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-729178ae-lpsdb"] Oct 30 16:40:06.395009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:06.394952 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f408729b-6480-4c89-8944-f9a7e1ab5736" path="/var/lib/kubelet/pods/f408729b-6480-4c89-8944-f9a7e1ab5736/volumes" Oct 30 16:40:07.030799 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:07.030757 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls"] Oct 30 16:40:07.042349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:07.042313 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-def4440f-r4bls"] Oct 30 16:40:08.395557 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:08.395524 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9ddd201-b315-49c5-83bf-6cacc3890c13" path="/var/lib/kubelet/pods/a9ddd201-b315-49c5-83bf-6cacc3890c13/volumes" Oct 30 16:40:12.028474 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:12.028436 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6"] Oct 30 16:40:12.040041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:12.039959 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31cb4ca-d6tn6"] Oct 30 16:40:12.395156 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:12.395074 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cc65f4c-eb79-43b2-b288-afae9f2a2fdf" path="/var/lib/kubelet/pods/3cc65f4c-eb79-43b2-b288-afae9f2a2fdf/volumes" Oct 30 16:40:15.564922 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:15.564879 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:19.030523 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:19.030489 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc"] Oct 30 16:40:19.041946 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:19.041909 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dbc862c-4ksrc"] Oct 30 16:40:20.394710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:20.394633 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305ae504-54ca-4dd1-9f5d-86fd928febf8" path="/var/lib/kubelet/pods/305ae504-54ca-4dd1-9f5d-86fd928febf8/volumes" Oct 30 16:40:23.285928 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.285890 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl"] Oct 30 16:40:23.291492 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.291472 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:23.309860 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.309826 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl"] Oct 30 16:40:23.316843 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.316815 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:40:23.316843 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.316821 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:40:23.375213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.375171 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:40:23.375213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.375212 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:40:23.393343 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.393311 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-zjwvl\" (UID: \"7ace71c4-cb6f-4835-8223-7832e17454f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:23.393500 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.393380 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-zjwvl\" (UID: \"7ace71c4-cb6f-4835-8223-7832e17454f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:23.393500 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.393480 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvn2t\" (UniqueName: \"kubernetes.io/projected/7ace71c4-cb6f-4835-8223-7832e17454f3-kube-api-access-dvn2t\") pod \"maas-api-revoke-sub-85237e45-zjwvl\" (UID: \"7ace71c4-cb6f-4835-8223-7832e17454f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:23.494787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.494745 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvn2t\" (UniqueName: \"kubernetes.io/projected/7ace71c4-cb6f-4835-8223-7832e17454f3-kube-api-access-dvn2t\") pod \"maas-api-revoke-sub-85237e45-zjwvl\" (UID: \"7ace71c4-cb6f-4835-8223-7832e17454f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:23.495018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.494822 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-zjwvl\" (UID: \"7ace71c4-cb6f-4835-8223-7832e17454f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:23.495018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.494984 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-zjwvl\" (UID: \"7ace71c4-cb6f-4835-8223-7832e17454f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:23.495557 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.495527 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-zjwvl\" (UID: \"7ace71c4-cb6f-4835-8223-7832e17454f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:23.497307 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.497283 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-zjwvl\" (UID: \"7ace71c4-cb6f-4835-8223-7832e17454f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:23.518097 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.518073 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvn2t\" (UniqueName: \"kubernetes.io/projected/7ace71c4-cb6f-4835-8223-7832e17454f3-kube-api-access-dvn2t\") pod \"maas-api-revoke-sub-85237e45-zjwvl\" (UID: \"7ace71c4-cb6f-4835-8223-7832e17454f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:23.605018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.604916 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:23.908578 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:23.908551 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl"] Oct 30 16:40:23.910595 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:40:23.910559 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ace71c4_cb6f_4835_8223_7832e17454f3.slice/crio-275891939c012af938beb2274b35b14bc4f88d8134958621dd2044af3b0dab98 WatchSource:0}: Error finding container 275891939c012af938beb2274b35b14bc4f88d8134958621dd2044af3b0dab98: Status 404 returned error can't find the container with id 275891939c012af938beb2274b35b14bc4f88d8134958621dd2044af3b0dab98 Oct 30 16:40:24.725162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:24.725126 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" event={"ID":"7ace71c4-cb6f-4835-8223-7832e17454f3","Type":"ContainerStarted","Data":"98ea9115b6e05b417f98cc68a655d49d9c7588bb7f401f312cbed6fc7f844992"} Oct 30 16:40:24.725656 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:24.725186 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" event={"ID":"7ace71c4-cb6f-4835-8223-7832e17454f3","Type":"ContainerStarted","Data":"275891939c012af938beb2274b35b14bc4f88d8134958621dd2044af3b0dab98"} Oct 30 16:40:24.767552 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:24.767493 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" podStartSLOduration=1.7674782439999999 podStartE2EDuration="1.767478244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:24.764538846 +0000 UTC m=+1186.513128255" watchObservedRunningTime="2026-10-30 16:40:24.767478244 +0000 UTC m=+1186.516067677" Oct 30 16:40:25.734422 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:25.734383 5838 generic.go:356] "Generic (PLEG): container finished" podID="7ace71c4-cb6f-4835-8223-7832e17454f3" containerID="98ea9115b6e05b417f98cc68a655d49d9c7588bb7f401f312cbed6fc7f844992" exitCode=0 Oct 30 16:40:25.734859 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:25.734450 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" event={"ID":"7ace71c4-cb6f-4835-8223-7832e17454f3","Type":"ContainerDied","Data":"98ea9115b6e05b417f98cc68a655d49d9c7588bb7f401f312cbed6fc7f844992"} Oct 30 16:40:26.878873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:26.878847 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:26.929875 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:26.929837 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-openshift-ca\") pod \"7ace71c4-cb6f-4835-8223-7832e17454f3\" (UID: \"7ace71c4-cb6f-4835-8223-7832e17454f3\") " Oct 30 16:40:26.930085 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:26.929906 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ace71c4-cb6f-4835-8223-7832e17454f3-kube-api-access-dvn2t\" (UniqueName: \"kubernetes.io/projected/7ace71c4-cb6f-4835-8223-7832e17454f3-kube-api-access-dvn2t\") pod \"7ace71c4-cb6f-4835-8223-7832e17454f3\" (UID: \"7ace71c4-cb6f-4835-8223-7832e17454f3\") " Oct 30 16:40:26.930085 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:26.930016 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-xks-ca\") pod \"7ace71c4-cb6f-4835-8223-7832e17454f3\" (UID: \"7ace71c4-cb6f-4835-8223-7832e17454f3\") " Oct 30 16:40:26.930291 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:26.930207 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-openshift-ca" pod "7ace71c4-cb6f-4835-8223-7832e17454f3" (UID: "7ace71c4-cb6f-4835-8223-7832e17454f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:26.930449 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:26.930429 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:40:26.932133 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:26.932108 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ace71c4-cb6f-4835-8223-7832e17454f3-kube-api-access-dvn2t" pod "7ace71c4-cb6f-4835-8223-7832e17454f3" (UID: "7ace71c4-cb6f-4835-8223-7832e17454f3"). InnerVolumeSpecName "kube-api-access-dvn2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:26.932236 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:26.932174 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-xks-ca" pod "7ace71c4-cb6f-4835-8223-7832e17454f3" (UID: "7ace71c4-cb6f-4835-8223-7832e17454f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:27.031795 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:27.031702 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvn2t\" (UniqueName: \"kubernetes.io/projected/7ace71c4-cb6f-4835-8223-7832e17454f3-kube-api-access-dvn2t\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:40:27.031795 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:27.031741 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ace71c4-cb6f-4835-8223-7832e17454f3-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:40:27.752384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:27.752351 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" Oct 30 16:40:27.752384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:27.752359 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl" event={"ID":"7ace71c4-cb6f-4835-8223-7832e17454f3","Type":"ContainerDied","Data":"275891939c012af938beb2274b35b14bc4f88d8134958621dd2044af3b0dab98"} Oct 30 16:40:27.752606 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:27.752420 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="275891939c012af938beb2274b35b14bc4f88d8134958621dd2044af3b0dab98" Oct 30 16:40:31.798186 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:40:31.798150 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:40:35.013826 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.013790 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85bb7dd889-q9jfl"] Oct 30 16:40:35.014567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.014037 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-85bb7dd889-q9jfl" podUID="a50d8645-b4aa-43f3-a079-f6c61a2bd38b" containerName="manager" containerID="cri-o://27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf" gracePeriod=30 Oct 30 16:40:35.265128 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.265073 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:35.414622 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.414586 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-metrics-certs\") pod \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\" (UID: \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\") " Oct 30 16:40:35.416645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.416597 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-webhook-cert\") pod \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\" (UID: \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\") " Oct 30 16:40:35.416835 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.416676 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-kube-api-access-qdvfr\" (UniqueName: \"kubernetes.io/projected/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-kube-api-access-qdvfr\") pod \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\" (UID: \"a50d8645-b4aa-43f3-a079-f6c61a2bd38b\") " Oct 30 16:40:35.418160 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.418127 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-metrics-certs" pod "a50d8645-b4aa-43f3-a079-f6c61a2bd38b" (UID: "a50d8645-b4aa-43f3-a079-f6c61a2bd38b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:35.418722 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.418701 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-webhook-cert" pod "a50d8645-b4aa-43f3-a079-f6c61a2bd38b" (UID: "a50d8645-b4aa-43f3-a079-f6c61a2bd38b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:35.418823 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.418798 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-kube-api-access-qdvfr" pod "a50d8645-b4aa-43f3-a079-f6c61a2bd38b" (UID: "a50d8645-b4aa-43f3-a079-f6c61a2bd38b"). InnerVolumeSpecName "kube-api-access-qdvfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:35.518451 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.518357 5838 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-webhook-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:40:35.518451 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.518387 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdvfr\" (UniqueName: \"kubernetes.io/projected/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-kube-api-access-qdvfr\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:40:35.518451 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.518396 5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a50d8645-b4aa-43f3-a079-f6c61a2bd38b-metrics-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:40:35.823364 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.823281 5838 generic.go:356] "Generic (PLEG): container finished" podID="a50d8645-b4aa-43f3-a079-f6c61a2bd38b" containerID="27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf" exitCode=0 Oct 30 16:40:35.823364 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.823328 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85bb7dd889-q9jfl" Oct 30 16:40:35.823551 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.823361 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85bb7dd889-q9jfl" event={"ID":"a50d8645-b4aa-43f3-a079-f6c61a2bd38b","Type":"ContainerDied","Data":"27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf"} Oct 30 16:40:35.823551 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.823419 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85bb7dd889-q9jfl" event={"ID":"a50d8645-b4aa-43f3-a079-f6c61a2bd38b","Type":"ContainerDied","Data":"3be1e02ed8ab32ef03a70e7ae06beca39c1f4a1ea86f35475f12f0cbd42a01ba"} Oct 30 16:40:35.823551 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.823450 5838 scope.go:122] "RemoveContainer" containerID="27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf" Oct 30 16:40:35.834680 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.834657 5838 scope.go:122] "RemoveContainer" containerID="27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf" Oct 30 16:40:35.835111 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:40:35.835090 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf\": container with ID starting with 27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf not found: ID does not exist" containerID="27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf" Oct 30 16:40:35.835180 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.835121 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf"} err="failed to get container status \"27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf\": rpc error: code = NotFound desc = could not find container \"27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf\": container with ID starting with 27541b3954af78ccd8f4d89cfdc9cb94d6f6746ec35b9910f5a35dc8e0f6a4cf not found: ID does not exist" Oct 30 16:40:35.873965 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.873932 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85bb7dd889-q9jfl"] Oct 30 16:40:35.884966 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:35.884937 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-85bb7dd889-q9jfl"] Oct 30 16:40:36.395119 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:36.395087 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50d8645-b4aa-43f3-a079-f6c61a2bd38b" path="/var/lib/kubelet/pods/a50d8645-b4aa-43f3-a079-f6c61a2bd38b/volumes" Oct 30 16:40:37.169372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.169338 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85bb7dd889-2mmgt"] Oct 30 16:40:37.169739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.169726 5838 cpu_manager.go:414] "containerMap: removing container" podUID="7ace71c4-cb6f-4835-8223-7832e17454f3" containerName="revoke-keys" Oct 30 16:40:37.169784 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.169740 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ace71c4-cb6f-4835-8223-7832e17454f3" containerName="revoke-keys" Oct 30 16:40:37.169784 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.169756 5838 cpu_manager.go:414] "containerMap: removing container" podUID="a50d8645-b4aa-43f3-a079-f6c61a2bd38b" containerName="manager" Oct 30 16:40:37.169784 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.169762 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50d8645-b4aa-43f3-a079-f6c61a2bd38b" containerName="manager" Oct 30 16:40:37.169880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.169853 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ace71c4-cb6f-4835-8223-7832e17454f3" containerName="revoke-keys" Oct 30 16:40:37.169880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.169862 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50d8645-b4aa-43f3-a079-f6c61a2bd38b" containerName="manager" Oct 30 16:40:37.173350 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.173325 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:40:37.189068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.189047 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 16:40:37.199840 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.199816 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lvg8k\"" Oct 30 16:40:37.199986 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.199816 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 16:40:37.199986 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.199815 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 16:40:37.204363 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.204341 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85bb7dd889-2mmgt"] Oct 30 16:40:37.333999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.333938 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gq7g\" (UniqueName: \"kubernetes.io/projected/fb90bed1-543d-499e-a996-85e01f7d3bcb-kube-api-access-4gq7g\") pod \"maas-controller-85bb7dd889-2mmgt\" (UID: \"fb90bed1-543d-499e-a996-85e01f7d3bcb\") " pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:40:37.334179 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.334134 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb90bed1-543d-499e-a996-85e01f7d3bcb-webhook-cert\") pod \"maas-controller-85bb7dd889-2mmgt\" (UID: \"fb90bed1-543d-499e-a996-85e01f7d3bcb\") " pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:40:37.334179 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.334168 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb90bed1-543d-499e-a996-85e01f7d3bcb-metrics-certs\") pod \"maas-controller-85bb7dd889-2mmgt\" (UID: \"fb90bed1-543d-499e-a996-85e01f7d3bcb\") " pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:40:37.435188 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.435107 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb90bed1-543d-499e-a996-85e01f7d3bcb-webhook-cert\") pod \"maas-controller-85bb7dd889-2mmgt\" (UID: \"fb90bed1-543d-499e-a996-85e01f7d3bcb\") " pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:40:37.435188 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.435150 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb90bed1-543d-499e-a996-85e01f7d3bcb-metrics-certs\") pod \"maas-controller-85bb7dd889-2mmgt\" (UID: \"fb90bed1-543d-499e-a996-85e01f7d3bcb\") " pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:40:37.435627 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.435190 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gq7g\" (UniqueName: \"kubernetes.io/projected/fb90bed1-543d-499e-a996-85e01f7d3bcb-kube-api-access-4gq7g\") pod \"maas-controller-85bb7dd889-2mmgt\" (UID: \"fb90bed1-543d-499e-a996-85e01f7d3bcb\") " pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:40:37.437714 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.437687 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb90bed1-543d-499e-a996-85e01f7d3bcb-webhook-cert\") pod \"maas-controller-85bb7dd889-2mmgt\" (UID: \"fb90bed1-543d-499e-a996-85e01f7d3bcb\") " pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:40:37.437830 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.437739 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb90bed1-543d-499e-a996-85e01f7d3bcb-metrics-certs\") pod \"maas-controller-85bb7dd889-2mmgt\" (UID: \"fb90bed1-543d-499e-a996-85e01f7d3bcb\") " pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:40:37.458144 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.458115 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gq7g\" (UniqueName: \"kubernetes.io/projected/fb90bed1-543d-499e-a996-85e01f7d3bcb-kube-api-access-4gq7g\") pod \"maas-controller-85bb7dd889-2mmgt\" (UID: \"fb90bed1-543d-499e-a996-85e01f7d3bcb\") " pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:40:37.486097 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.486063 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:40:37.789512 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.789488 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85bb7dd889-2mmgt"] Oct 30 16:40:37.792147 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:40:37.792120 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb90bed1_543d_499e_a996_85e01f7d3bcb.slice/crio-c7e703b7943e723a6060b2d4404d211a7b02289101ef58605ed5a29a0b2ecbbb WatchSource:0}: Error finding container c7e703b7943e723a6060b2d4404d211a7b02289101ef58605ed5a29a0b2ecbbb: Status 404 returned error can't find the container with id c7e703b7943e723a6060b2d4404d211a7b02289101ef58605ed5a29a0b2ecbbb Oct 30 16:40:37.842406 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:37.842368 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85bb7dd889-2mmgt" event={"ID":"fb90bed1-543d-499e-a996-85e01f7d3bcb","Type":"ContainerStarted","Data":"c7e703b7943e723a6060b2d4404d211a7b02289101ef58605ed5a29a0b2ecbbb"} Oct 30 16:40:38.571079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:38.571050 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:40:38.575150 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:38.575129 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:40:38.851797 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:38.851703 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85bb7dd889-2mmgt" event={"ID":"fb90bed1-543d-499e-a996-85e01f7d3bcb","Type":"ContainerStarted","Data":"6dde7b046e158076a612e66b602537505851e9634748b9e20ea462940985d148"} Oct 30 16:40:38.852006 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:38.851957 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:40:38.895457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:38.895404 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-85bb7dd889-2mmgt" podStartSLOduration=1.5939495510000001 podStartE2EDuration="1.895391677s" totalImagesPullingTime="301.442126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:38.891957967 +0000 UTC m=+1200.640547354" watchObservedRunningTime="2026-10-30 16:40:38.895391677 +0000 UTC m=+1200.643981063" Oct 30 16:40:39.881455 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:39.881425 5838 scope.go:122] "RemoveContainer" containerID="89de5e902858c80419923098cb7441cdfff779c808f5fa2e920ca033c77a95c5" Oct 30 16:40:39.896222 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:39.896197 5838 scope.go:122] "RemoveContainer" containerID="f482b6908622e2ebb3b16c5751540e2eec15e35346f33ef0326d3d045d83b467" Oct 30 16:40:39.905611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:39.905589 5838 scope.go:122] "RemoveContainer" containerID="abe068da2accb199d935f0cfc0c1fec21d4d40737353f6c049e448eca4cd3322" Oct 30 16:40:39.914958 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:39.914941 5838 scope.go:122] "RemoveContainer" containerID="248464c2db6ecb65dbe2bf6bb79fd9c086ed6ad7f19e16149335281fd1cde605" Oct 30 16:40:39.923704 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:39.923686 5838 scope.go:122] "RemoveContainer" containerID="d0dbe610ef6b1d5e1d1f1efb14bbe5f9a97b782aada55d569dab4828dbb9ec2e" Oct 30 16:40:49.863245 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:40:49.863210 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85bb7dd889-2mmgt" Oct 30 16:41:01.796910 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:41:01.796865 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:41:05.925392 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:05.925357 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:41:11.707601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.707564 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q"] Oct 30 16:41:11.711669 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.711638 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:11.731103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.731075 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q"] Oct 30 16:41:11.738119 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.738083 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:41:11.738119 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.738109 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:41:11.738309 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.738209 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:41:11.738309 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.738275 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:41:11.784275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.784243 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kfl8q\" (UID: \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:11.784466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.784372 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng65f\" (UniqueName: \"kubernetes.io/projected/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-kube-api-access-ng65f\") pod \"maas-api-revoke-sub-c1e271a9-kfl8q\" (UID: \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:11.784466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.784446 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kfl8q\" (UID: \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:11.885996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.885934 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng65f\" (UniqueName: \"kubernetes.io/projected/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-kube-api-access-ng65f\") pod \"maas-api-revoke-sub-c1e271a9-kfl8q\" (UID: \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:11.885996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.886001 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kfl8q\" (UID: \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:11.886263 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.886056 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kfl8q\" (UID: \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:11.886710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.886689 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kfl8q\" (UID: \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:11.888385 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.888364 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kfl8q\" (UID: \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:11.908794 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:11.908764 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng65f\" (UniqueName: \"kubernetes.io/projected/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-kube-api-access-ng65f\") pod \"maas-api-revoke-sub-c1e271a9-kfl8q\" (UID: \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:12.024811 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:12.024777 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:12.327617 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:12.327588 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q"] Oct 30 16:41:12.329410 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:41:12.329380 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d607990_f7c6_4317_9a6c_a3b1d22fc0a8.slice/crio-6bf9665c6154676712e89b0fa7ebc56199bfa6a71d78be406ce1e20347d9587f WatchSource:0}: Error finding container 6bf9665c6154676712e89b0fa7ebc56199bfa6a71d78be406ce1e20347d9587f: Status 404 returned error can't find the container with id 6bf9665c6154676712e89b0fa7ebc56199bfa6a71d78be406ce1e20347d9587f Oct 30 16:41:13.127939 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:13.127900 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" event={"ID":"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8","Type":"ContainerStarted","Data":"6735d54ac5a4dda0807f6e9fc7e1b26520c2cfb1679c8411ec382c9007bcae87"} Oct 30 16:41:13.128444 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:13.127952 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" event={"ID":"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8","Type":"ContainerStarted","Data":"6bf9665c6154676712e89b0fa7ebc56199bfa6a71d78be406ce1e20347d9587f"} Oct 30 16:41:13.169051 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:13.168776 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" podStartSLOduration=2.16876019 podStartE2EDuration="2.16876019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:13.166554215 +0000 UTC m=+1234.915143603" watchObservedRunningTime="2026-10-30 16:41:13.16876019 +0000 UTC m=+1234.917349576" Oct 30 16:41:14.135954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:14.135910 5838 generic.go:356] "Generic (PLEG): container finished" podID="6d607990-f7c6-4317-9a6c-a3b1d22fc0a8" containerID="6735d54ac5a4dda0807f6e9fc7e1b26520c2cfb1679c8411ec382c9007bcae87" exitCode=0 Oct 30 16:41:14.136387 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:14.136011 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" event={"ID":"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8","Type":"ContainerDied","Data":"6735d54ac5a4dda0807f6e9fc7e1b26520c2cfb1679c8411ec382c9007bcae87"} Oct 30 16:41:15.289996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:15.289952 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:15.423984 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:15.423891 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-xks-ca\") pod \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\" (UID: \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\") " Oct 30 16:41:15.424143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:15.424027 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-openshift-ca\") pod \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\" (UID: \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\") " Oct 30 16:41:15.424143 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:15.424083 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-kube-api-access-ng65f\" (UniqueName: \"kubernetes.io/projected/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-kube-api-access-ng65f\") pod \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\" (UID: \"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8\") " Oct 30 16:41:15.424379 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:15.424344 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-openshift-ca" pod "6d607990-f7c6-4317-9a6c-a3b1d22fc0a8" (UID: "6d607990-f7c6-4317-9a6c-a3b1d22fc0a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:15.426200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:15.426177 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-kube-api-access-ng65f" pod "6d607990-f7c6-4317-9a6c-a3b1d22fc0a8" (UID: "6d607990-f7c6-4317-9a6c-a3b1d22fc0a8"). InnerVolumeSpecName "kube-api-access-ng65f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:15.426294 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:15.426238 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-xks-ca" pod "6d607990-f7c6-4317-9a6c-a3b1d22fc0a8" (UID: "6d607990-f7c6-4317-9a6c-a3b1d22fc0a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:15.525818 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:15.525772 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:41:15.525818 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:15.525817 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:41:15.525818 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:15.525832 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng65f\" (UniqueName: \"kubernetes.io/projected/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8-kube-api-access-ng65f\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:41:16.153486 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:16.153443 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" Oct 30 16:41:16.153696 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:16.153464 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q" event={"ID":"6d607990-f7c6-4317-9a6c-a3b1d22fc0a8","Type":"ContainerDied","Data":"6bf9665c6154676712e89b0fa7ebc56199bfa6a71d78be406ce1e20347d9587f"} Oct 30 16:41:16.153696 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:16.153538 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bf9665c6154676712e89b0fa7ebc56199bfa6a71d78be406ce1e20347d9587f" Oct 30 16:41:18.035829 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:18.035792 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr"] Oct 30 16:41:18.047389 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:18.047356 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4dfefcc-lgkmr"] Oct 30 16:41:18.395642 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:18.395560 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987d59bf-7b46-45de-a4da-00d2e913b3b9" path="/var/lib/kubelet/pods/987d59bf-7b46-45de-a4da-00d2e913b3b9/volumes" Oct 30 16:41:22.029493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:22.029456 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b"] Oct 30 16:41:22.040929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:22.040889 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e901e7a-z559b"] Oct 30 16:41:22.394789 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:22.394710 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b65095-6bd0-449b-9051-4e77c282b043" path="/var/lib/kubelet/pods/c3b65095-6bd0-449b-9051-4e77c282b043/volumes" Oct 30 16:41:22.957157 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:22.957118 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:41:24.040773 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:24.040729 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb"] Oct 30 16:41:24.052433 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:24.052395 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4"] Oct 30 16:41:24.055817 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:24.055792 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d38b37f4-k79wb"] Oct 30 16:41:24.058782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:24.058754 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffa68cb-8lsq4"] Oct 30 16:41:24.395229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:24.395151 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bd21e5-a642-4ea3-a0ac-9f640ef94625" path="/var/lib/kubelet/pods/43bd21e5-a642-4ea3-a0ac-9f640ef94625/volumes" Oct 30 16:41:24.395598 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:24.395584 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd3df30-43f9-46f3-9be2-b24b2b021395" path="/var/lib/kubelet/pods/abd3df30-43f9-46f3-9be2-b24b2b021395/volumes" Oct 30 16:41:27.241633 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:27.241598 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:41:31.797853 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:41:31.797811 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:41:31.855936 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.855895 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w"] Oct 30 16:41:31.856372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.856357 5838 cpu_manager.go:414] "containerMap: removing container" podUID="6d607990-f7c6-4317-9a6c-a3b1d22fc0a8" containerName="revoke-keys" Oct 30 16:41:31.856417 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.856373 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d607990-f7c6-4317-9a6c-a3b1d22fc0a8" containerName="revoke-keys" Oct 30 16:41:31.856473 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.856463 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d607990-f7c6-4317-9a6c-a3b1d22fc0a8" containerName="revoke-keys" Oct 30 16:41:31.859429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.859411 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:31.886292 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.886258 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:41:31.886292 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.886284 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:41:31.886531 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.886328 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:41:31.886531 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.886378 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:41:31.890049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.890016 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w"] Oct 30 16:41:31.907273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.907237 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4p24\" (UniqueName: \"kubernetes.io/projected/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-kube-api-access-w4p24\") pod \"maas-api-revoke-sub-2e1f5b5f-j7j7w\" (UID: \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:31.907453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.907287 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-j7j7w\" (UID: \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:31.907453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:31.907316 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-j7j7w\" (UID: \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:32.008237 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:32.008197 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4p24\" (UniqueName: \"kubernetes.io/projected/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-kube-api-access-w4p24\") pod \"maas-api-revoke-sub-2e1f5b5f-j7j7w\" (UID: \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:32.008429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:32.008257 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-j7j7w\" (UID: \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:32.008429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:32.008290 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-j7j7w\" (UID: \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:32.009073 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:32.009044 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-j7j7w\" (UID: \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:32.010822 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:32.010795 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-j7j7w\" (UID: \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:32.029777 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:32.029738 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl"] Oct 30 16:41:32.032662 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:32.032626 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5eb544e-56hzl"] Oct 30 16:41:32.033227 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:32.033202 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4p24\" (UniqueName: \"kubernetes.io/projected/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-kube-api-access-w4p24\") pod \"maas-api-revoke-sub-2e1f5b5f-j7j7w\" (UID: \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:32.173266 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:32.173179 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:32.396416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:32.396058 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeaf9129-0999-46d3-801c-1bb480135c6f" path="/var/lib/kubelet/pods/aeaf9129-0999-46d3-801c-1bb480135c6f/volumes" Oct 30 16:41:32.477923 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:32.477896 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w"] Oct 30 16:41:32.480364 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:41:32.480328 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7dd338_d668_4bd3_8fa3_41a8ab062a8c.slice/crio-f7415a69a1d152c8574556a87632319a7fa2018aa786fe53930885a6b9e0cddd WatchSource:0}: Error finding container f7415a69a1d152c8574556a87632319a7fa2018aa786fe53930885a6b9e0cddd: Status 404 returned error can't find the container with id f7415a69a1d152c8574556a87632319a7fa2018aa786fe53930885a6b9e0cddd Oct 30 16:41:33.298952 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:33.298908 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" event={"ID":"da7dd338-d668-4bd3-8fa3-41a8ab062a8c","Type":"ContainerStarted","Data":"2f81f9ff5b444ac7c4981ee5e18637aeec192c4b40567c8f404068a6d2f49ad9"} Oct 30 16:41:33.299393 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:33.298992 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" event={"ID":"da7dd338-d668-4bd3-8fa3-41a8ab062a8c","Type":"ContainerStarted","Data":"f7415a69a1d152c8574556a87632319a7fa2018aa786fe53930885a6b9e0cddd"} Oct 30 16:41:33.358999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:33.358925 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" podStartSLOduration=2.358908943 podStartE2EDuration="2.358908943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:33.355764724 +0000 UTC m=+1255.104354124" watchObservedRunningTime="2026-10-30 16:41:33.358908943 +0000 UTC m=+1255.107498329" Oct 30 16:41:34.307184 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:34.307148 5838 generic.go:356] "Generic (PLEG): container finished" podID="da7dd338-d668-4bd3-8fa3-41a8ab062a8c" containerID="2f81f9ff5b444ac7c4981ee5e18637aeec192c4b40567c8f404068a6d2f49ad9" exitCode=0 Oct 30 16:41:34.307582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:34.307239 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" event={"ID":"da7dd338-d668-4bd3-8fa3-41a8ab062a8c","Type":"ContainerDied","Data":"2f81f9ff5b444ac7c4981ee5e18637aeec192c4b40567c8f404068a6d2f49ad9"} Oct 30 16:41:35.450888 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:35.450858 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:35.545465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:35.545424 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-xks-ca\") pod \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\" (UID: \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\") " Oct 30 16:41:35.545647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:35.545515 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-openshift-ca\") pod \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\" (UID: \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\") " Oct 30 16:41:35.545647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:35.545625 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-kube-api-access-w4p24\" (UniqueName: \"kubernetes.io/projected/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-kube-api-access-w4p24\") pod \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\" (UID: \"da7dd338-d668-4bd3-8fa3-41a8ab062a8c\") " Oct 30 16:41:35.545871 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:35.545849 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-openshift-ca" pod "da7dd338-d668-4bd3-8fa3-41a8ab062a8c" (UID: "da7dd338-d668-4bd3-8fa3-41a8ab062a8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:35.547653 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:35.547622 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-kube-api-access-w4p24" pod "da7dd338-d668-4bd3-8fa3-41a8ab062a8c" (UID: "da7dd338-d668-4bd3-8fa3-41a8ab062a8c"). InnerVolumeSpecName "kube-api-access-w4p24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:35.547782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:35.547681 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-xks-ca" pod "da7dd338-d668-4bd3-8fa3-41a8ab062a8c" (UID: "da7dd338-d668-4bd3-8fa3-41a8ab062a8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:35.647378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:35.647280 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:41:35.647378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:35.647314 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4p24\" (UniqueName: \"kubernetes.io/projected/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-kube-api-access-w4p24\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:41:35.647378 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:35.647324 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7dd338-d668-4bd3-8fa3-41a8ab062a8c-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:41:36.029389 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:36.029349 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5"] Oct 30 16:41:36.040877 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:36.040840 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6624684f-t8px5"] Oct 30 16:41:36.322787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:36.322709 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" Oct 30 16:41:36.322787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:36.322745 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w" event={"ID":"da7dd338-d668-4bd3-8fa3-41a8ab062a8c","Type":"ContainerDied","Data":"f7415a69a1d152c8574556a87632319a7fa2018aa786fe53930885a6b9e0cddd"} Oct 30 16:41:36.322988 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:36.322795 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7415a69a1d152c8574556a87632319a7fa2018aa786fe53930885a6b9e0cddd" Oct 30 16:41:36.394790 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:36.394757 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a8b00a-4d8e-4428-80eb-b651e333a38f" path="/var/lib/kubelet/pods/48a8b00a-4d8e-4428-80eb-b651e333a38f/volumes" Oct 30 16:41:37.032062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:37.032027 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9"] Oct 30 16:41:37.043260 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:37.043225 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d117a55f-kppf9"] Oct 30 16:41:38.403089 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:38.403059 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e2aba0-8b73-4995-950e-6408c1e6f10f" path="/var/lib/kubelet/pods/19e2aba0-8b73-4995-950e-6408c1e6f10f/volumes" Oct 30 16:41:39.031705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:39.031669 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f"] Oct 30 16:41:39.043133 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:39.043092 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c2159f-sht4f"] Oct 30 16:41:40.072064 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:40.072026 5838 scope.go:122] "RemoveContainer" containerID="5f35db3f1cdf9a1d75fa6e035c35249d958afb4cc35808d53b5c4ba95e15571f" Oct 30 16:41:40.082639 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:40.082617 5838 scope.go:122] "RemoveContainer" containerID="3f413fbe8428acb53a7bc2747e8435378a0efc099acb1d06ebe2400cfaeef290" Oct 30 16:41:40.092289 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:40.092268 5838 scope.go:122] "RemoveContainer" containerID="033ff8103126d53c0641dec71490c0e4a656ebca34442fa6ffd235f70dd49e06" Oct 30 16:41:40.103953 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:40.103929 5838 scope.go:122] "RemoveContainer" containerID="72f7efe879e2e2e6554c3b0abff52f20ad3b2e578d829aaec00b998baf9b64e1" Oct 30 16:41:40.112778 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:40.112758 5838 scope.go:122] "RemoveContainer" containerID="fa48448a73bb9556de3bfaf09ea2a81d217e143ccf675af91c42a49f0ae8ae51" Oct 30 16:41:40.121666 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:40.121644 5838 scope.go:122] "RemoveContainer" containerID="961b28dda32ad29a2f3d5468f507e7d481e93f721917cbd6dd36e35efb977e51" Oct 30 16:41:40.130212 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:40.130191 5838 scope.go:122] "RemoveContainer" containerID="36c40e0bd903c3df8108b7b44969ab276cb211e5337c30e6abd0c5456dff1f72" Oct 30 16:41:40.395083 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:40.395003 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995c7339-905b-459a-88bf-9971b42ca95b" path="/var/lib/kubelet/pods/995c7339-905b-459a-88bf-9971b42ca95b/volumes" Oct 30 16:41:43.038599 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.038563 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:41:43.230341 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.230303 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp"] Oct 30 16:41:43.230725 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.230711 5838 cpu_manager.go:414] "containerMap: removing container" podUID="da7dd338-d668-4bd3-8fa3-41a8ab062a8c" containerName="revoke-keys" Oct 30 16:41:43.230767 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.230726 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7dd338-d668-4bd3-8fa3-41a8ab062a8c" containerName="revoke-keys" Oct 30 16:41:43.230813 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.230804 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7dd338-d668-4bd3-8fa3-41a8ab062a8c" containerName="revoke-keys" Oct 30 16:41:43.238986 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.238927 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:43.254270 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.254217 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:41:43.254479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.254449 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:41:43.254633 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.254614 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:41:43.256114 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.256089 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp"] Oct 30 16:41:43.261356 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.261330 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:41:43.318853 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.318760 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h97gf\" (UniqueName: \"kubernetes.io/projected/e2329056-283e-4b01-8b5c-61c3ef288f99-kube-api-access-h97gf\") pod \"maas-api-revoke-sub-ec843018-4bcmp\" (UID: \"e2329056-283e-4b01-8b5c-61c3ef288f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:43.319038 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.318883 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4bcmp\" (UID: \"e2329056-283e-4b01-8b5c-61c3ef288f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:43.319038 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.318939 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4bcmp\" (UID: \"e2329056-283e-4b01-8b5c-61c3ef288f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:43.420193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.420159 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4bcmp\" (UID: \"e2329056-283e-4b01-8b5c-61c3ef288f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:43.420375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.420217 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4bcmp\" (UID: \"e2329056-283e-4b01-8b5c-61c3ef288f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:43.420375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.420272 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h97gf\" (UniqueName: \"kubernetes.io/projected/e2329056-283e-4b01-8b5c-61c3ef288f99-kube-api-access-h97gf\") pod \"maas-api-revoke-sub-ec843018-4bcmp\" (UID: \"e2329056-283e-4b01-8b5c-61c3ef288f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:43.420847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.420824 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4bcmp\" (UID: \"e2329056-283e-4b01-8b5c-61c3ef288f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:43.422744 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.422658 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4bcmp\" (UID: \"e2329056-283e-4b01-8b5c-61c3ef288f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:43.449888 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.449816 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h97gf\" (UniqueName: \"kubernetes.io/projected/e2329056-283e-4b01-8b5c-61c3ef288f99-kube-api-access-h97gf\") pod \"maas-api-revoke-sub-ec843018-4bcmp\" (UID: \"e2329056-283e-4b01-8b5c-61c3ef288f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:43.554187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.554151 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:43.862039 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:43.861939 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp"] Oct 30 16:41:43.864789 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:41:43.864755 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2329056_283e_4b01_8b5c_61c3ef288f99.slice/crio-1e7e1e678669ac7b26a7eb156c14371124ac1c3fff64db1eff695d62fd876cca WatchSource:0}: Error finding container 1e7e1e678669ac7b26a7eb156c14371124ac1c3fff64db1eff695d62fd876cca: Status 404 returned error can't find the container with id 1e7e1e678669ac7b26a7eb156c14371124ac1c3fff64db1eff695d62fd876cca Oct 30 16:41:44.381085 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:44.381049 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" event={"ID":"e2329056-283e-4b01-8b5c-61c3ef288f99","Type":"ContainerStarted","Data":"b862db8c4825d8bfe518c27ccb5ecfd91ccc3c3eccdbeae630e45604725f98b1"} Oct 30 16:41:44.381602 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:44.381110 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" event={"ID":"e2329056-283e-4b01-8b5c-61c3ef288f99","Type":"ContainerStarted","Data":"1e7e1e678669ac7b26a7eb156c14371124ac1c3fff64db1eff695d62fd876cca"} Oct 30 16:41:44.420608 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:44.420542 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" podStartSLOduration=1.420523861 podStartE2EDuration="1.420523861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:44.420097593 +0000 UTC m=+1266.168686979" watchObservedRunningTime="2026-10-30 16:41:44.420523861 +0000 UTC m=+1266.169113248" Oct 30 16:41:45.392249 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:45.392217 5838 generic.go:356] "Generic (PLEG): container finished" podID="e2329056-283e-4b01-8b5c-61c3ef288f99" containerID="b862db8c4825d8bfe518c27ccb5ecfd91ccc3c3eccdbeae630e45604725f98b1" exitCode=0 Oct 30 16:41:45.392649 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:45.392258 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" event={"ID":"e2329056-283e-4b01-8b5c-61c3ef288f99","Type":"ContainerDied","Data":"b862db8c4825d8bfe518c27ccb5ecfd91ccc3c3eccdbeae630e45604725f98b1"} Oct 30 16:41:46.561963 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:46.561932 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:46.654923 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:46.654866 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2329056-283e-4b01-8b5c-61c3ef288f99-kube-api-access-h97gf\" (UniqueName: \"kubernetes.io/projected/e2329056-283e-4b01-8b5c-61c3ef288f99-kube-api-access-h97gf\") pod \"e2329056-283e-4b01-8b5c-61c3ef288f99\" (UID: \"e2329056-283e-4b01-8b5c-61c3ef288f99\") " Oct 30 16:41:46.655160 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:46.654940 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-openshift-ca\") pod \"e2329056-283e-4b01-8b5c-61c3ef288f99\" (UID: \"e2329056-283e-4b01-8b5c-61c3ef288f99\") " Oct 30 16:41:46.655160 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:46.655067 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-xks-ca\") pod \"e2329056-283e-4b01-8b5c-61c3ef288f99\" (UID: \"e2329056-283e-4b01-8b5c-61c3ef288f99\") " Oct 30 16:41:46.655745 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:46.655703 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-openshift-ca" pod "e2329056-283e-4b01-8b5c-61c3ef288f99" (UID: "e2329056-283e-4b01-8b5c-61c3ef288f99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:46.658079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:46.658050 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-xks-ca" pod "e2329056-283e-4b01-8b5c-61c3ef288f99" (UID: "e2329056-283e-4b01-8b5c-61c3ef288f99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:46.658361 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:46.658305 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2329056-283e-4b01-8b5c-61c3ef288f99-kube-api-access-h97gf" pod "e2329056-283e-4b01-8b5c-61c3ef288f99" (UID: "e2329056-283e-4b01-8b5c-61c3ef288f99"). InnerVolumeSpecName "kube-api-access-h97gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:46.756996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:46.756578 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h97gf\" (UniqueName: \"kubernetes.io/projected/e2329056-283e-4b01-8b5c-61c3ef288f99-kube-api-access-h97gf\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:41:46.756996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:46.756625 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:41:46.756996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:46.756643 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2329056-283e-4b01-8b5c-61c3ef288f99-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:41:47.066081 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:47.066041 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf"] Oct 30 16:41:47.069549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:47.069516 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86b26dcf-gqzkf"] Oct 30 16:41:47.409898 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:47.409811 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" event={"ID":"e2329056-283e-4b01-8b5c-61c3ef288f99","Type":"ContainerDied","Data":"1e7e1e678669ac7b26a7eb156c14371124ac1c3fff64db1eff695d62fd876cca"} Oct 30 16:41:47.409898 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:47.409864 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e7e1e678669ac7b26a7eb156c14371124ac1c3fff64db1eff695d62fd876cca" Oct 30 16:41:47.409898 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:47.409827 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp" Oct 30 16:41:48.401997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:48.401063 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856dbe15-b87c-4c29-9b17-33107352e240" path="/var/lib/kubelet/pods/856dbe15-b87c-4c29-9b17-33107352e240/volumes" Oct 30 16:41:48.507686 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:48.507129 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:41:48.507686 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:48.507206 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-djmdh" podUID="3465e7a5-d6cc-4c8c-9052-29034ba0847f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:51.028634 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:51.028557 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq"] Oct 30 16:41:51.040300 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:51.040268 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2"] Oct 30 16:41:51.043234 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:51.043208 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f819f0-zktnq"] Oct 30 16:41:51.045878 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:51.045855 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a67b072-5vcw2"] Oct 30 16:41:52.395086 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:52.395053 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2019e57f-4469-4d96-882a-88f69dfbb5b3" path="/var/lib/kubelet/pods/2019e57f-4469-4d96-882a-88f69dfbb5b3/volumes" Oct 30 16:41:52.395533 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:52.395518 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01b5da0-655c-4866-b665-5acac0fa0fee" path="/var/lib/kubelet/pods/b01b5da0-655c-4866-b665-5acac0fa0fee/volumes" Oct 30 16:41:55.031722 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:55.031678 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf"] Oct 30 16:41:55.043368 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:55.043337 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe785e9-95dxf"] Oct 30 16:41:56.394700 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:56.394663 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2807e2f-a164-4293-bf8b-4461980d31ec" path="/var/lib/kubelet/pods/d2807e2f-a164-4293-bf8b-4461980d31ec/volumes" Oct 30 16:41:57.341123 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:41:57.341087 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:42:01.795808 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:42:01.795758 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:42:10.032923 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.032884 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7"] Oct 30 16:42:10.043518 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.043483 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74447167-w5hw7"] Oct 30 16:42:10.226475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.226438 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm"] Oct 30 16:42:10.226856 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.226843 5838 cpu_manager.go:414] "containerMap: removing container" podUID="e2329056-283e-4b01-8b5c-61c3ef288f99" containerName="revoke-keys" Oct 30 16:42:10.226902 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.226857 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2329056-283e-4b01-8b5c-61c3ef288f99" containerName="revoke-keys" Oct 30 16:42:10.226952 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.226943 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2329056-283e-4b01-8b5c-61c3ef288f99" containerName="revoke-keys" Oct 30 16:42:10.230343 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.230326 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:10.246267 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.246239 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:10.246267 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.246268 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:42:10.246475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.246338 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:10.246475 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.246357 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:10.249877 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.249852 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm"] Oct 30 16:42:10.313228 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.313146 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpz2t\" (UniqueName: \"kubernetes.io/projected/d5246c38-187e-4be4-ad87-990d95716d1f-kube-api-access-hpz2t\") pod \"maas-api-revoke-sub-046e1343-b7mtm\" (UID: \"d5246c38-187e-4be4-ad87-990d95716d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:10.313228 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.313189 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-b7mtm\" (UID: \"d5246c38-187e-4be4-ad87-990d95716d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:10.313437 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.313283 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-b7mtm\" (UID: \"d5246c38-187e-4be4-ad87-990d95716d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:10.394027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.393992 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d56ec6-bce3-46c4-b471-e1c5cfe006e4" path="/var/lib/kubelet/pods/84d56ec6-bce3-46c4-b471-e1c5cfe006e4/volumes" Oct 30 16:42:10.414241 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.414213 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpz2t\" (UniqueName: \"kubernetes.io/projected/d5246c38-187e-4be4-ad87-990d95716d1f-kube-api-access-hpz2t\") pod \"maas-api-revoke-sub-046e1343-b7mtm\" (UID: \"d5246c38-187e-4be4-ad87-990d95716d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:10.414387 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.414253 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-b7mtm\" (UID: \"d5246c38-187e-4be4-ad87-990d95716d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:10.414387 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.414288 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-b7mtm\" (UID: \"d5246c38-187e-4be4-ad87-990d95716d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:10.414918 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.414898 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-b7mtm\" (UID: \"d5246c38-187e-4be4-ad87-990d95716d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:10.416592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.416549 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-b7mtm\" (UID: \"d5246c38-187e-4be4-ad87-990d95716d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:10.437097 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.437073 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpz2t\" (UniqueName: \"kubernetes.io/projected/d5246c38-187e-4be4-ad87-990d95716d1f-kube-api-access-hpz2t\") pod \"maas-api-revoke-sub-046e1343-b7mtm\" (UID: \"d5246c38-187e-4be4-ad87-990d95716d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:10.543344 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.543307 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:10.850127 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:10.849993 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm"] Oct 30 16:42:10.853131 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:42:10.853102 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5246c38_187e_4be4_ad87_990d95716d1f.slice/crio-99b33a23459c67bf5a26c0c0002911fe7704d7f5ea301ca578f22ef19da091d6 WatchSource:0}: Error finding container 99b33a23459c67bf5a26c0c0002911fe7704d7f5ea301ca578f22ef19da091d6: Status 404 returned error can't find the container with id 99b33a23459c67bf5a26c0c0002911fe7704d7f5ea301ca578f22ef19da091d6 Oct 30 16:42:11.597902 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:11.597861 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" event={"ID":"d5246c38-187e-4be4-ad87-990d95716d1f","Type":"ContainerStarted","Data":"2822164b8c3ccd1e7813b0e349cf0416339fa5ab499de82a9b13c2ff60b426fb"} Oct 30 16:42:11.598341 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:11.597923 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" event={"ID":"d5246c38-187e-4be4-ad87-990d95716d1f","Type":"ContainerStarted","Data":"99b33a23459c67bf5a26c0c0002911fe7704d7f5ea301ca578f22ef19da091d6"} Oct 30 16:42:11.639646 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:11.639588 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" podStartSLOduration=1.6395705459999999 podStartE2EDuration="1.639570546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:11.637628701 +0000 UTC m=+1293.386218094" watchObservedRunningTime="2026-10-30 16:42:11.639570546 +0000 UTC m=+1293.388159933" Oct 30 16:42:12.606442 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:12.606409 5838 generic.go:356] "Generic (PLEG): container finished" podID="d5246c38-187e-4be4-ad87-990d95716d1f" containerID="2822164b8c3ccd1e7813b0e349cf0416339fa5ab499de82a9b13c2ff60b426fb" exitCode=0 Oct 30 16:42:12.606833 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:12.606484 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" event={"ID":"d5246c38-187e-4be4-ad87-990d95716d1f","Type":"ContainerDied","Data":"2822164b8c3ccd1e7813b0e349cf0416339fa5ab499de82a9b13c2ff60b426fb"} Oct 30 16:42:13.748710 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:13.748684 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:13.845479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:13.845445 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-xks-ca\") pod \"d5246c38-187e-4be4-ad87-990d95716d1f\" (UID: \"d5246c38-187e-4be4-ad87-990d95716d1f\") " Oct 30 16:42:13.845653 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:13.845490 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5246c38-187e-4be4-ad87-990d95716d1f-kube-api-access-hpz2t\" (UniqueName: \"kubernetes.io/projected/d5246c38-187e-4be4-ad87-990d95716d1f-kube-api-access-hpz2t\") pod \"d5246c38-187e-4be4-ad87-990d95716d1f\" (UID: \"d5246c38-187e-4be4-ad87-990d95716d1f\") " Oct 30 16:42:13.845653 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:13.845589 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-openshift-ca\") pod \"d5246c38-187e-4be4-ad87-990d95716d1f\" (UID: \"d5246c38-187e-4be4-ad87-990d95716d1f\") " Oct 30 16:42:13.845965 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:13.845940 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-openshift-ca" pod "d5246c38-187e-4be4-ad87-990d95716d1f" (UID: "d5246c38-187e-4be4-ad87-990d95716d1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:13.847739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:13.847709 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5246c38-187e-4be4-ad87-990d95716d1f-kube-api-access-hpz2t" pod "d5246c38-187e-4be4-ad87-990d95716d1f" (UID: "d5246c38-187e-4be4-ad87-990d95716d1f"). InnerVolumeSpecName "kube-api-access-hpz2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:13.847739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:13.847716 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-xks-ca" pod "d5246c38-187e-4be4-ad87-990d95716d1f" (UID: "d5246c38-187e-4be4-ad87-990d95716d1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:13.947491 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:13.947405 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:42:13.947491 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:13.947436 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpz2t\" (UniqueName: \"kubernetes.io/projected/d5246c38-187e-4be4-ad87-990d95716d1f-kube-api-access-hpz2t\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:42:13.947491 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:13.947446 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5246c38-187e-4be4-ad87-990d95716d1f-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:42:14.027661 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:14.027624 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d"] Oct 30 16:42:14.030760 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:14.030734 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47170442-ndz4d"] Oct 30 16:42:14.394132 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:14.394100 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5" path="/var/lib/kubelet/pods/21e5ff25-1c4b-42ed-aa82-d2b34dbfd6e5/volumes" Oct 30 16:42:14.622155 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:14.622125 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" Oct 30 16:42:14.622327 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:14.622159 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm" event={"ID":"d5246c38-187e-4be4-ad87-990d95716d1f","Type":"ContainerDied","Data":"99b33a23459c67bf5a26c0c0002911fe7704d7f5ea301ca578f22ef19da091d6"} Oct 30 16:42:14.622327 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:14.622207 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99b33a23459c67bf5a26c0c0002911fe7704d7f5ea301ca578f22ef19da091d6" Oct 30 16:42:20.560355 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.560314 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75"] Oct 30 16:42:20.560761 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.560747 5838 cpu_manager.go:414] "containerMap: removing container" podUID="d5246c38-187e-4be4-ad87-990d95716d1f" containerName="revoke-keys" Oct 30 16:42:20.560806 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.560761 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5246c38-187e-4be4-ad87-990d95716d1f" containerName="revoke-keys" Oct 30 16:42:20.560846 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.560827 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5246c38-187e-4be4-ad87-990d95716d1f" containerName="revoke-keys" Oct 30 16:42:20.565252 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.565230 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:20.584353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.584314 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75"] Oct 30 16:42:20.591679 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.591648 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:20.591927 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.591683 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:20.591927 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.591688 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:20.592109 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.591735 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:42:20.712806 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.712765 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-9qn75\" (UID: \"a204b7f6-1772-45f7-a2de-946dd200d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:20.713028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.712835 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjtll\" (UniqueName: \"kubernetes.io/projected/a204b7f6-1772-45f7-a2de-946dd200d134-kube-api-access-pjtll\") pod \"maas-api-revoke-sub-d3fba056-9qn75\" (UID: \"a204b7f6-1772-45f7-a2de-946dd200d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:20.713028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.712932 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-9qn75\" (UID: \"a204b7f6-1772-45f7-a2de-946dd200d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:20.813700 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.813615 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-9qn75\" (UID: \"a204b7f6-1772-45f7-a2de-946dd200d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:20.813700 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.813687 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjtll\" (UniqueName: \"kubernetes.io/projected/a204b7f6-1772-45f7-a2de-946dd200d134-kube-api-access-pjtll\") pod \"maas-api-revoke-sub-d3fba056-9qn75\" (UID: \"a204b7f6-1772-45f7-a2de-946dd200d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:20.813927 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.813732 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-9qn75\" (UID: \"a204b7f6-1772-45f7-a2de-946dd200d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:20.814497 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.814469 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-9qn75\" (UID: \"a204b7f6-1772-45f7-a2de-946dd200d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:20.816019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.815993 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-9qn75\" (UID: \"a204b7f6-1772-45f7-a2de-946dd200d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:20.837286 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.837253 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjtll\" (UniqueName: \"kubernetes.io/projected/a204b7f6-1772-45f7-a2de-946dd200d134-kube-api-access-pjtll\") pod \"maas-api-revoke-sub-d3fba056-9qn75\" (UID: \"a204b7f6-1772-45f7-a2de-946dd200d134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:20.878393 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:20.878352 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:21.182477 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:21.182445 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75"] Oct 30 16:42:21.185442 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:42:21.185404 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda204b7f6_1772_45f7_a2de_946dd200d134.slice/crio-be4356ef7981bbae8e908faca83678da75a8b176e77a8b8e752d1236bec871c1 WatchSource:0}: Error finding container be4356ef7981bbae8e908faca83678da75a8b176e77a8b8e752d1236bec871c1: Status 404 returned error can't find the container with id be4356ef7981bbae8e908faca83678da75a8b176e77a8b8e752d1236bec871c1 Oct 30 16:42:21.520729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:21.520690 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:42:21.672705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:21.672667 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" event={"ID":"a204b7f6-1772-45f7-a2de-946dd200d134","Type":"ContainerStarted","Data":"695ffb70044859ef31831115596eed0787148d58f673a6f7b8c18327d69f0efd"} Oct 30 16:42:21.673144 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:21.672726 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" event={"ID":"a204b7f6-1772-45f7-a2de-946dd200d134","Type":"ContainerStarted","Data":"be4356ef7981bbae8e908faca83678da75a8b176e77a8b8e752d1236bec871c1"} Oct 30 16:42:21.714282 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:21.714165 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" podStartSLOduration=1.7141491850000001 podStartE2EDuration="1.714149185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:21.712481455 +0000 UTC m=+1303.461070843" watchObservedRunningTime="2026-10-30 16:42:21.714149185 +0000 UTC m=+1303.462738569" Oct 30 16:42:22.685056 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:22.685009 5838 generic.go:356] "Generic (PLEG): container finished" podID="a204b7f6-1772-45f7-a2de-946dd200d134" containerID="695ffb70044859ef31831115596eed0787148d58f673a6f7b8c18327d69f0efd" exitCode=0 Oct 30 16:42:22.685512 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:22.685064 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" event={"ID":"a204b7f6-1772-45f7-a2de-946dd200d134","Type":"ContainerDied","Data":"695ffb70044859ef31831115596eed0787148d58f673a6f7b8c18327d69f0efd"} Oct 30 16:42:24.085773 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.085739 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:24.098148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.098115 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-openshift-ca\") pod \"a204b7f6-1772-45f7-a2de-946dd200d134\" (UID: \"a204b7f6-1772-45f7-a2de-946dd200d134\") " Oct 30 16:42:24.098539 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.098513 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-openshift-ca" pod "a204b7f6-1772-45f7-a2de-946dd200d134" (UID: "a204b7f6-1772-45f7-a2de-946dd200d134"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:24.098691 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.098650 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a204b7f6-1772-45f7-a2de-946dd200d134-kube-api-access-pjtll\" (UniqueName: \"kubernetes.io/projected/a204b7f6-1772-45f7-a2de-946dd200d134-kube-api-access-pjtll\") pod \"a204b7f6-1772-45f7-a2de-946dd200d134\" (UID: \"a204b7f6-1772-45f7-a2de-946dd200d134\") " Oct 30 16:42:24.099105 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.099083 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-xks-ca\") pod \"a204b7f6-1772-45f7-a2de-946dd200d134\" (UID: \"a204b7f6-1772-45f7-a2de-946dd200d134\") " Oct 30 16:42:24.099512 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.099490 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:42:24.101076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.101052 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a204b7f6-1772-45f7-a2de-946dd200d134-kube-api-access-pjtll" pod "a204b7f6-1772-45f7-a2de-946dd200d134" (UID: "a204b7f6-1772-45f7-a2de-946dd200d134"). InnerVolumeSpecName "kube-api-access-pjtll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:24.101557 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.101529 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-xks-ca" pod "a204b7f6-1772-45f7-a2de-946dd200d134" (UID: "a204b7f6-1772-45f7-a2de-946dd200d134"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:24.200893 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.200850 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjtll\" (UniqueName: \"kubernetes.io/projected/a204b7f6-1772-45f7-a2de-946dd200d134-kube-api-access-pjtll\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:42:24.200893 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.200887 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a204b7f6-1772-45f7-a2de-946dd200d134-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:42:24.703583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.703493 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" Oct 30 16:42:24.703583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.703528 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75" event={"ID":"a204b7f6-1772-45f7-a2de-946dd200d134","Type":"ContainerDied","Data":"be4356ef7981bbae8e908faca83678da75a8b176e77a8b8e752d1236bec871c1"} Oct 30 16:42:24.703583 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:24.703573 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be4356ef7981bbae8e908faca83678da75a8b176e77a8b8e752d1236bec871c1" Oct 30 16:42:31.797069 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:42:31.797032 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:42:31.822659 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:31.822622 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:42:36.331724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:36.331683 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:42:40.185745 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:40.185714 5838 scope.go:122] "RemoveContainer" containerID="4c6cafa548e2a41bddffa3284b7c1704dcc1d4a5220e7d9736f2962137a10f7a" Oct 30 16:42:40.198239 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:40.198209 5838 scope.go:122] "RemoveContainer" containerID="6fb8c29dfdb12c47ce69dc1a9c5af09ee8aefed892857da5f719e2f471b9ada4" Oct 30 16:42:40.207706 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:40.207686 5838 scope.go:122] "RemoveContainer" containerID="0cfdc5a8b5a72b790fe6c7a1aa761aad1d3db5a2e9ba20d6968f2e61f95e66fc" Oct 30 16:42:40.216724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:40.216698 5838 scope.go:122] "RemoveContainer" containerID="5150e66809fbbdc9de891f7fe17d072a334f30efc0188a3f413fc94abe2733a4" Oct 30 16:42:40.225467 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:40.225441 5838 scope.go:122] "RemoveContainer" containerID="f07ecf60957a8a5bc7b723f2b33700dc1202e74db1e5512b314bd53b17436b53" Oct 30 16:42:40.234820 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:40.234799 5838 scope.go:122] "RemoveContainer" containerID="ec5f874975d96ff2692b0c1f102f1bf8f5bace786b377befac3e0066f79a81b6" Oct 30 16:42:40.243479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:40.243456 5838 scope.go:122] "RemoveContainer" containerID="fc099bcad56c75368b3cc5062ef324efe31283732aaa6aafd0d687aca7fa619b" Oct 30 16:42:40.252268 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:40.252249 5838 scope.go:122] "RemoveContainer" containerID="738034b92a1bc701fbb71e860be89f69f2469ab2515bc379bde68ea4af0ffc70" Oct 30 16:42:41.027871 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:41.027835 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp"] Oct 30 16:42:41.039408 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:41.039375 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bbe1b9f-xcvlp"] Oct 30 16:42:42.394695 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:42.394659 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c612c8-2f17-4a5e-88f5-4927cd45e4cd" path="/var/lib/kubelet/pods/b9c612c8-2f17-4a5e-88f5-4927cd45e4cd/volumes" Oct 30 16:42:42.907302 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:42.907271 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r"] Oct 30 16:42:42.907774 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:42.907754 5838 cpu_manager.go:414] "containerMap: removing container" podUID="a204b7f6-1772-45f7-a2de-946dd200d134" containerName="revoke-keys" Oct 30 16:42:42.907884 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:42.907775 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a204b7f6-1772-45f7-a2de-946dd200d134" containerName="revoke-keys" Oct 30 16:42:42.907950 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:42.907930 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a204b7f6-1772-45f7-a2de-946dd200d134" containerName="revoke-keys" Oct 30 16:42:42.913096 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:42.913072 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:42.930173 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:42.930143 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r"] Oct 30 16:42:42.938171 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:42.938144 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:42.938171 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:42.938148 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:42:42.938402 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:42.938195 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:42.938581 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:42.938559 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:43.014159 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.014122 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6vn7r\" (UID: \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:43.014159 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.014161 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmxzh\" (UniqueName: \"kubernetes.io/projected/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-kube-api-access-cmxzh\") pod \"maas-api-revoke-sub-ae594979-6vn7r\" (UID: \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:43.014362 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.014200 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6vn7r\" (UID: \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:43.115166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.115115 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6vn7r\" (UID: \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:43.115166 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.115170 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmxzh\" (UniqueName: \"kubernetes.io/projected/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-kube-api-access-cmxzh\") pod \"maas-api-revoke-sub-ae594979-6vn7r\" (UID: \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:43.115432 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.115224 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6vn7r\" (UID: \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:43.116371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.116341 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6vn7r\" (UID: \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:43.125935 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.125906 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6vn7r\" (UID: \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:43.138689 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.138658 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmxzh\" (UniqueName: \"kubernetes.io/projected/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-kube-api-access-cmxzh\") pod \"maas-api-revoke-sub-ae594979-6vn7r\" (UID: \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:43.226088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.226008 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:43.527257 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.527232 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r"] Oct 30 16:42:43.530079 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:42:43.530051 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff72534d_dc5b_4a4c_8489_6650f0ba6a8f.slice/crio-b58cf71a41f9321ab7c5d35fdd49513c77fb1e6acb991ce7403334bc68e189f6 WatchSource:0}: Error finding container b58cf71a41f9321ab7c5d35fdd49513c77fb1e6acb991ce7403334bc68e189f6: Status 404 returned error can't find the container with id b58cf71a41f9321ab7c5d35fdd49513c77fb1e6acb991ce7403334bc68e189f6 Oct 30 16:42:43.842857 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.842820 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" event={"ID":"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f","Type":"ContainerStarted","Data":"7b18bcc2919f2e3e7bb19cf8e788045cc8fa5052e94e52ac2df5580014dc101b"} Oct 30 16:42:43.843031 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.842886 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" event={"ID":"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f","Type":"ContainerStarted","Data":"b58cf71a41f9321ab7c5d35fdd49513c77fb1e6acb991ce7403334bc68e189f6"} Oct 30 16:42:43.882375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:43.882317 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" podStartSLOduration=1.8823031879999998 podStartE2EDuration="1.882303188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:43.881214727 +0000 UTC m=+1325.629804114" watchObservedRunningTime="2026-10-30 16:42:43.882303188 +0000 UTC m=+1325.630892574" Oct 30 16:42:44.850805 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:44.850769 5838 generic.go:356] "Generic (PLEG): container finished" podID="ff72534d-dc5b-4a4c-8489-6650f0ba6a8f" containerID="7b18bcc2919f2e3e7bb19cf8e788045cc8fa5052e94e52ac2df5580014dc101b" exitCode=0 Oct 30 16:42:44.851228 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:44.850859 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" event={"ID":"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f","Type":"ContainerDied","Data":"7b18bcc2919f2e3e7bb19cf8e788045cc8fa5052e94e52ac2df5580014dc101b"} Oct 30 16:42:45.989103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:45.989079 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:46.144896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:46.144794 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-xks-ca\") pod \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\" (UID: \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\") " Oct 30 16:42:46.144896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:46.144861 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-openshift-ca\") pod \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\" (UID: \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\") " Oct 30 16:42:46.145140 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:46.144964 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-kube-api-access-cmxzh\" (UniqueName: \"kubernetes.io/projected/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-kube-api-access-cmxzh\") pod \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\" (UID: \"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f\") " Oct 30 16:42:46.145261 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:46.145239 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-openshift-ca" pod "ff72534d-dc5b-4a4c-8489-6650f0ba6a8f" (UID: "ff72534d-dc5b-4a4c-8489-6650f0ba6a8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:46.147024 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:46.147000 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-xks-ca" pod "ff72534d-dc5b-4a4c-8489-6650f0ba6a8f" (UID: "ff72534d-dc5b-4a4c-8489-6650f0ba6a8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:46.147154 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:46.147133 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-kube-api-access-cmxzh" pod "ff72534d-dc5b-4a4c-8489-6650f0ba6a8f" (UID: "ff72534d-dc5b-4a4c-8489-6650f0ba6a8f"). InnerVolumeSpecName "kube-api-access-cmxzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:46.246270 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:46.246233 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:42:46.246270 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:46.246266 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:42:46.246448 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:46.246278 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmxzh\" (UniqueName: \"kubernetes.io/projected/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f-kube-api-access-cmxzh\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:42:46.866031 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:46.865997 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" Oct 30 16:42:46.866215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:46.865999 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r" event={"ID":"ff72534d-dc5b-4a4c-8489-6650f0ba6a8f","Type":"ContainerDied","Data":"b58cf71a41f9321ab7c5d35fdd49513c77fb1e6acb991ce7403334bc68e189f6"} Oct 30 16:42:46.866215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:46.866124 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b58cf71a41f9321ab7c5d35fdd49513c77fb1e6acb991ce7403334bc68e189f6" Oct 30 16:42:51.029985 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:51.029935 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct"] Oct 30 16:42:51.041279 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:51.041251 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a065f3-xcxct"] Oct 30 16:42:52.394720 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:52.394688 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18151a38-458f-4a95-a49f-19ee96bf9c52" path="/var/lib/kubelet/pods/18151a38-458f-4a95-a49f-19ee96bf9c52/volumes" Oct 30 16:42:54.213938 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:54.213903 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:42:55.029022 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:55.028986 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2"] Oct 30 16:42:55.040348 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:55.040316 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57857170-t48k2"] Oct 30 16:42:56.394855 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:56.394820 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4697e804-bb04-4e91-a607-3be930573237" path="/var/lib/kubelet/pods/4697e804-bb04-4e91-a607-3be930573237/volumes" Oct 30 16:42:58.953665 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:42:58.953610 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:43:01.796518 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:43:01.796475 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:43:04.615266 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.615225 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47"] Oct 30 16:43:04.615758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.615651 5838 cpu_manager.go:414] "containerMap: removing container" podUID="ff72534d-dc5b-4a4c-8489-6650f0ba6a8f" containerName="revoke-keys" Oct 30 16:43:04.615758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.615662 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff72534d-dc5b-4a4c-8489-6650f0ba6a8f" containerName="revoke-keys" Oct 30 16:43:04.615758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.615750 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff72534d-dc5b-4a4c-8489-6650f0ba6a8f" containerName="revoke-keys" Oct 30 16:43:04.619124 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.619104 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:04.638282 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.638247 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47"] Oct 30 16:43:04.646313 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.646277 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:43:04.646313 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.646310 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:43:04.646507 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.646284 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:43:04.646507 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.646378 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:43:04.648881 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.648851 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7sn47\" (UID: \"b72042b0-a36d-4079-9d08-712865ac0660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:04.648964 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.648903 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clhkl\" (UniqueName: \"kubernetes.io/projected/b72042b0-a36d-4079-9d08-712865ac0660-kube-api-access-clhkl\") pod \"maas-api-revoke-sub-a4aef7d7-7sn47\" (UID: \"b72042b0-a36d-4079-9d08-712865ac0660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:04.649047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.649028 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7sn47\" (UID: \"b72042b0-a36d-4079-9d08-712865ac0660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:04.750328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.750278 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7sn47\" (UID: \"b72042b0-a36d-4079-9d08-712865ac0660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:04.750559 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.750428 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7sn47\" (UID: \"b72042b0-a36d-4079-9d08-712865ac0660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:04.750559 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.750463 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clhkl\" (UniqueName: \"kubernetes.io/projected/b72042b0-a36d-4079-9d08-712865ac0660-kube-api-access-clhkl\") pod \"maas-api-revoke-sub-a4aef7d7-7sn47\" (UID: \"b72042b0-a36d-4079-9d08-712865ac0660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:04.751103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.751077 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7sn47\" (UID: \"b72042b0-a36d-4079-9d08-712865ac0660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:04.752787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.752760 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7sn47\" (UID: \"b72042b0-a36d-4079-9d08-712865ac0660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:04.773644 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.773605 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clhkl\" (UniqueName: \"kubernetes.io/projected/b72042b0-a36d-4079-9d08-712865ac0660-kube-api-access-clhkl\") pod \"maas-api-revoke-sub-a4aef7d7-7sn47\" (UID: \"b72042b0-a36d-4079-9d08-712865ac0660\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:04.933017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:04.932894 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:05.238517 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:05.238490 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47"] Oct 30 16:43:05.241068 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:43:05.241038 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72042b0_a36d_4079_9d08_712865ac0660.slice/crio-f9fa7d647691a2dbe642b39d54ef91574054b40d42c6b8322ea7deff18bd0527 WatchSource:0}: Error finding container f9fa7d647691a2dbe642b39d54ef91574054b40d42c6b8322ea7deff18bd0527: Status 404 returned error can't find the container with id f9fa7d647691a2dbe642b39d54ef91574054b40d42c6b8322ea7deff18bd0527 Oct 30 16:43:06.031113 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:06.031075 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" event={"ID":"b72042b0-a36d-4079-9d08-712865ac0660","Type":"ContainerStarted","Data":"9a9dd273f44e04acd3ec82a9a0e90d6e27d4048f0047f7a0162cefede0ed0910"} Oct 30 16:43:06.031499 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:06.031161 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" event={"ID":"b72042b0-a36d-4079-9d08-712865ac0660","Type":"ContainerStarted","Data":"f9fa7d647691a2dbe642b39d54ef91574054b40d42c6b8322ea7deff18bd0527"} Oct 30 16:43:06.070434 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:06.070384 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" podStartSLOduration=2.070368604 podStartE2EDuration="2.070368604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:06.069372542 +0000 UTC m=+1347.817961928" watchObservedRunningTime="2026-10-30 16:43:06.070368604 +0000 UTC m=+1347.818957990" Oct 30 16:43:07.039029 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:07.038988 5838 generic.go:356] "Generic (PLEG): container finished" podID="b72042b0-a36d-4079-9d08-712865ac0660" containerID="9a9dd273f44e04acd3ec82a9a0e90d6e27d4048f0047f7a0162cefede0ed0910" exitCode=0 Oct 30 16:43:07.039435 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:07.039061 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" event={"ID":"b72042b0-a36d-4079-9d08-712865ac0660","Type":"ContainerDied","Data":"9a9dd273f44e04acd3ec82a9a0e90d6e27d4048f0047f7a0162cefede0ed0910"} Oct 30 16:43:08.181044 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:08.181017 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:08.284131 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:08.284088 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72042b0-a36d-4079-9d08-712865ac0660-kube-api-access-clhkl\" (UniqueName: \"kubernetes.io/projected/b72042b0-a36d-4079-9d08-712865ac0660-kube-api-access-clhkl\") pod \"b72042b0-a36d-4079-9d08-712865ac0660\" (UID: \"b72042b0-a36d-4079-9d08-712865ac0660\") " Oct 30 16:43:08.284316 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:08.284149 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-xks-ca\") pod \"b72042b0-a36d-4079-9d08-712865ac0660\" (UID: \"b72042b0-a36d-4079-9d08-712865ac0660\") " Oct 30 16:43:08.284316 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:08.284179 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-openshift-ca\") pod \"b72042b0-a36d-4079-9d08-712865ac0660\" (UID: \"b72042b0-a36d-4079-9d08-712865ac0660\") " Oct 30 16:43:08.284647 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:08.284623 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-openshift-ca" pod "b72042b0-a36d-4079-9d08-712865ac0660" (UID: "b72042b0-a36d-4079-9d08-712865ac0660"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:08.286333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:08.286311 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-xks-ca" pod "b72042b0-a36d-4079-9d08-712865ac0660" (UID: "b72042b0-a36d-4079-9d08-712865ac0660"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:08.286424 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:08.286404 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72042b0-a36d-4079-9d08-712865ac0660-kube-api-access-clhkl" pod "b72042b0-a36d-4079-9d08-712865ac0660" (UID: "b72042b0-a36d-4079-9d08-712865ac0660"). InnerVolumeSpecName "kube-api-access-clhkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:08.385964 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:08.385877 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clhkl\" (UniqueName: \"kubernetes.io/projected/b72042b0-a36d-4079-9d08-712865ac0660-kube-api-access-clhkl\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:43:08.385964 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:08.385906 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:43:08.385964 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:08.385917 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b72042b0-a36d-4079-9d08-712865ac0660-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:43:09.053536 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:09.053505 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" Oct 30 16:43:09.053536 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:09.053529 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47" event={"ID":"b72042b0-a36d-4079-9d08-712865ac0660","Type":"ContainerDied","Data":"f9fa7d647691a2dbe642b39d54ef91574054b40d42c6b8322ea7deff18bd0527"} Oct 30 16:43:09.053760 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:09.053579 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9fa7d647691a2dbe642b39d54ef91574054b40d42c6b8322ea7deff18bd0527" Oct 30 16:43:15.028848 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:15.028816 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5"] Oct 30 16:43:15.040397 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:15.040365 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0075b4bf-hczr5"] Oct 30 16:43:15.811386 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:15.811351 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:43:16.394592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:16.394560 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b0a141-4f14-4f9a-8d19-7267172c99ae" path="/var/lib/kubelet/pods/97b0a141-4f14-4f9a-8d19-7267172c99ae/volumes" Oct 30 16:43:20.945246 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:20.945162 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:43:25.029120 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:25.029081 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h"] Oct 30 16:43:25.040398 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:25.040365 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd"] Oct 30 16:43:25.043598 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:25.043574 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aeddee8-bsg9h"] Oct 30 16:43:25.045959 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:25.045933 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e604791-dxxwd"] Oct 30 16:43:26.395332 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:26.395300 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2bfb675-2581-4e1b-a305-215a157b86dd" path="/var/lib/kubelet/pods/a2bfb675-2581-4e1b-a305-215a157b86dd/volumes" Oct 30 16:43:26.395767 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:26.395751 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af" path="/var/lib/kubelet/pods/f2418c9f-2c7e-4ed8-9c3e-443c88a6c6af/volumes" Oct 30 16:43:27.039572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.039540 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8"] Oct 30 16:43:27.050433 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.050402 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30626ac9-56ht8"] Oct 30 16:43:27.282414 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.282369 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn"] Oct 30 16:43:27.282817 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.282803 5838 cpu_manager.go:414] "containerMap: removing container" podUID="b72042b0-a36d-4079-9d08-712865ac0660" containerName="revoke-keys" Oct 30 16:43:27.282873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.282818 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72042b0-a36d-4079-9d08-712865ac0660" containerName="revoke-keys" Oct 30 16:43:27.282908 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.282897 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72042b0-a36d-4079-9d08-712865ac0660" containerName="revoke-keys" Oct 30 16:43:27.287683 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.287661 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:27.313808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.313777 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:43:27.314033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.313840 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:43:27.314033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.313850 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:43:27.314033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.313944 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:43:27.317013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.316987 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn"] Oct 30 16:43:27.410431 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.410396 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wntvv\" (UniqueName: \"kubernetes.io/projected/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-kube-api-access-wntvv\") pod \"maas-api-revoke-sub-79805ca9-h66vn\" (UID: \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:27.410929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.410516 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-h66vn\" (UID: \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:27.410929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.410576 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-h66vn\" (UID: \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:27.511347 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.511307 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wntvv\" (UniqueName: \"kubernetes.io/projected/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-kube-api-access-wntvv\") pod \"maas-api-revoke-sub-79805ca9-h66vn\" (UID: \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:27.511528 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.511401 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-h66vn\" (UID: \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:27.511528 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.511433 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-h66vn\" (UID: \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:27.512137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.512114 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-h66vn\" (UID: \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:27.513885 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.513863 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-h66vn\" (UID: \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:27.534802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.534770 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wntvv\" (UniqueName: \"kubernetes.io/projected/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-kube-api-access-wntvv\") pod \"maas-api-revoke-sub-79805ca9-h66vn\" (UID: \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:27.601243 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.601169 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:27.904886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:27.904810 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn"] Oct 30 16:43:28.204262 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:28.204177 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" event={"ID":"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba","Type":"ContainerStarted","Data":"7822f48efe098ea9e9490ac3977e943b8a5ee9214736011b855343c1672db5fa"} Oct 30 16:43:28.395037 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:28.395001 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa987b7-701a-47b6-8c36-d59787846fbb" path="/var/lib/kubelet/pods/6aa987b7-701a-47b6-8c36-d59787846fbb/volumes" Oct 30 16:43:29.212388 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:29.212295 5838 generic.go:356] "Generic (PLEG): container finished" podID="dcf3a0e2-29e7-4080-9f88-8da63aaa25ba" containerID="da9f3216f716411ee60a4cc91243483793509b858b5c212495dbf49ff16a3f9f" exitCode=0 Oct 30 16:43:29.212388 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:29.212335 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" event={"ID":"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba","Type":"ContainerDied","Data":"da9f3216f716411ee60a4cc91243483793509b858b5c212495dbf49ff16a3f9f"} Oct 30 16:43:30.355741 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:30.355711 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:30.437195 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:30.437161 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-openshift-ca\") pod \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\" (UID: \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\") " Oct 30 16:43:30.437363 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:30.437212 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-xks-ca\") pod \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\" (UID: \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\") " Oct 30 16:43:30.437363 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:30.437313 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-kube-api-access-wntvv\" (UniqueName: \"kubernetes.io/projected/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-kube-api-access-wntvv\") pod \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\" (UID: \"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba\") " Oct 30 16:43:30.437547 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:30.437523 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-openshift-ca" pod "dcf3a0e2-29e7-4080-9f88-8da63aaa25ba" (UID: "dcf3a0e2-29e7-4080-9f88-8da63aaa25ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:30.439394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:30.439365 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-xks-ca" pod "dcf3a0e2-29e7-4080-9f88-8da63aaa25ba" (UID: "dcf3a0e2-29e7-4080-9f88-8da63aaa25ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:30.439539 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:30.439396 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-kube-api-access-wntvv" pod "dcf3a0e2-29e7-4080-9f88-8da63aaa25ba" (UID: "dcf3a0e2-29e7-4080-9f88-8da63aaa25ba"). InnerVolumeSpecName "kube-api-access-wntvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:30.538486 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:30.538454 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:43:30.538486 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:30.538484 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wntvv\" (UniqueName: \"kubernetes.io/projected/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-kube-api-access-wntvv\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:43:30.538700 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:30.538495 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:43:31.226155 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:31.226127 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" Oct 30 16:43:31.226334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:31.226161 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn" event={"ID":"dcf3a0e2-29e7-4080-9f88-8da63aaa25ba","Type":"ContainerDied","Data":"7822f48efe098ea9e9490ac3977e943b8a5ee9214736011b855343c1672db5fa"} Oct 30 16:43:31.226334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:31.226211 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7822f48efe098ea9e9490ac3977e943b8a5ee9214736011b855343c1672db5fa" Oct 30 16:43:31.796622 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:43:31.796583 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:43:39.420150 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:39.420113 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:43:40.309979 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:40.309930 5838 scope.go:122] "RemoveContainer" containerID="47031d2482b741a596bb9ed448d7dd7707fe4fd89f1cfd4a01312c1e2dca1493" Oct 30 16:43:40.322622 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:40.322596 5838 scope.go:122] "RemoveContainer" containerID="0d6ba2ce285c3177590370ab5c21a447dd8f80806a80431ca2cb2f6d3bd72c93" Oct 30 16:43:40.334389 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:40.334364 5838 scope.go:122] "RemoveContainer" containerID="91e701d09ad297cf7a5566d60e6d8ec252939b1fba5283c3df317ced0ddaab8f" Oct 30 16:43:40.344917 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:40.344891 5838 scope.go:122] "RemoveContainer" containerID="c60942bab1ffb4e5ae96ae6db4c87bf41182eb944e4a14436b2dea0f9bbac377" Oct 30 16:43:40.356742 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:40.356715 5838 scope.go:122] "RemoveContainer" containerID="1a23f0c2a06c8900664355c92733cbe5e9d14ff492d1d63af06bdcff1f9e1bd5" Oct 30 16:43:40.367096 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:40.367073 5838 scope.go:122] "RemoveContainer" containerID="737ce2dcfea75c397dd118e2f9cfeafd3b87d03f42dccd45e32f9736ee9706dd" Oct 30 16:43:40.377933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:40.377911 5838 scope.go:122] "RemoveContainer" containerID="f4130a9bf6a46d00fffe0c7636be1ec0bf5443b16a28bcafec92801ef34839d7" Oct 30 16:43:44.029596 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:44.029554 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755"] Oct 30 16:43:44.041032 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:44.040994 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b89a3a85-vr755"] Oct 30 16:43:44.395152 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:44.395074 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0113a6c2-db8d-469c-aefa-0d8b5a9e321a" path="/var/lib/kubelet/pods/0113a6c2-db8d-469c-aefa-0d8b5a9e321a/volumes" Oct 30 16:43:45.198904 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.198864 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j"] Oct 30 16:43:45.199342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.199303 5838 cpu_manager.go:414] "containerMap: removing container" podUID="dcf3a0e2-29e7-4080-9f88-8da63aaa25ba" containerName="revoke-keys" Oct 30 16:43:45.199342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.199315 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf3a0e2-29e7-4080-9f88-8da63aaa25ba" containerName="revoke-keys" Oct 30 16:43:45.199415 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.199392 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf3a0e2-29e7-4080-9f88-8da63aaa25ba" containerName="revoke-keys" Oct 30 16:43:45.204620 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.204599 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:45.222130 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.222092 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j"] Oct 30 16:43:45.230220 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.230187 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:43:45.230220 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.230208 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:43:45.230220 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.230193 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:43:45.230468 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.230263 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:43:45.319292 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.319260 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s4m2j\" (UID: \"97eb915b-40dc-4ab9-bde8-3410417ca78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:45.319471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.319420 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s4m2j\" (UID: \"97eb915b-40dc-4ab9-bde8-3410417ca78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:45.319471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.319453 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w2bp\" (UniqueName: \"kubernetes.io/projected/97eb915b-40dc-4ab9-bde8-3410417ca78f-kube-api-access-2w2bp\") pod \"maas-api-revoke-sub-0b7e8edb-s4m2j\" (UID: \"97eb915b-40dc-4ab9-bde8-3410417ca78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:45.420895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.420862 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s4m2j\" (UID: \"97eb915b-40dc-4ab9-bde8-3410417ca78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:45.420895 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.420900 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w2bp\" (UniqueName: \"kubernetes.io/projected/97eb915b-40dc-4ab9-bde8-3410417ca78f-kube-api-access-2w2bp\") pod \"maas-api-revoke-sub-0b7e8edb-s4m2j\" (UID: \"97eb915b-40dc-4ab9-bde8-3410417ca78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:45.421193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.420931 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s4m2j\" (UID: \"97eb915b-40dc-4ab9-bde8-3410417ca78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:45.421621 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.421598 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s4m2j\" (UID: \"97eb915b-40dc-4ab9-bde8-3410417ca78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:45.423272 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.423246 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s4m2j\" (UID: \"97eb915b-40dc-4ab9-bde8-3410417ca78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:45.446503 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.446476 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w2bp\" (UniqueName: \"kubernetes.io/projected/97eb915b-40dc-4ab9-bde8-3410417ca78f-kube-api-access-2w2bp\") pod \"maas-api-revoke-sub-0b7e8edb-s4m2j\" (UID: \"97eb915b-40dc-4ab9-bde8-3410417ca78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:45.517579 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.517548 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:45.819396 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:45.819368 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j"] Oct 30 16:43:45.822238 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:43:45.822205 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97eb915b_40dc_4ab9_bde8_3410417ca78f.slice/crio-23e0008ac2a3c78884e43a592a31e4c7941d4f203b03008083fbdc704d0f1696 WatchSource:0}: Error finding container 23e0008ac2a3c78884e43a592a31e4c7941d4f203b03008083fbdc704d0f1696: Status 404 returned error can't find the container with id 23e0008ac2a3c78884e43a592a31e4c7941d4f203b03008083fbdc704d0f1696 Oct 30 16:43:46.028417 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:46.028379 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn"] Oct 30 16:43:46.031275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:46.031243 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec92c54e-cmknn"] Oct 30 16:43:46.333063 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:46.333019 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" event={"ID":"97eb915b-40dc-4ab9-bde8-3410417ca78f","Type":"ContainerStarted","Data":"b385730b4a5b1c4fa48365c629e97b9902e2a45d8c7dacb8bea19cd13a57f9d3"} Oct 30 16:43:46.333455 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:46.333087 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" event={"ID":"97eb915b-40dc-4ab9-bde8-3410417ca78f","Type":"ContainerStarted","Data":"23e0008ac2a3c78884e43a592a31e4c7941d4f203b03008083fbdc704d0f1696"} Oct 30 16:43:46.373506 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:46.373388 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" podStartSLOduration=1.3733727390000001 podStartE2EDuration="1.373372739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:46.370632769 +0000 UTC m=+1388.119222156" watchObservedRunningTime="2026-10-30 16:43:46.373372739 +0000 UTC m=+1388.121962126" Oct 30 16:43:46.394480 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:46.394443 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243" path="/var/lib/kubelet/pods/9fc01ad1-1d4f-4ac5-8b7b-d7c582e13243/volumes" Oct 30 16:43:47.341642 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:47.341607 5838 generic.go:356] "Generic (PLEG): container finished" podID="97eb915b-40dc-4ab9-bde8-3410417ca78f" containerID="b385730b4a5b1c4fa48365c629e97b9902e2a45d8c7dacb8bea19cd13a57f9d3" exitCode=0 Oct 30 16:43:47.342063 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:47.341682 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" event={"ID":"97eb915b-40dc-4ab9-bde8-3410417ca78f","Type":"ContainerDied","Data":"b385730b4a5b1c4fa48365c629e97b9902e2a45d8c7dacb8bea19cd13a57f9d3"} Oct 30 16:43:48.481084 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:48.481056 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:48.549893 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:48.549856 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-openshift-ca\") pod \"97eb915b-40dc-4ab9-bde8-3410417ca78f\" (UID: \"97eb915b-40dc-4ab9-bde8-3410417ca78f\") " Oct 30 16:43:48.550103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:48.549951 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-xks-ca\") pod \"97eb915b-40dc-4ab9-bde8-3410417ca78f\" (UID: \"97eb915b-40dc-4ab9-bde8-3410417ca78f\") " Oct 30 16:43:48.550103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:48.550006 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97eb915b-40dc-4ab9-bde8-3410417ca78f-kube-api-access-2w2bp\" (UniqueName: \"kubernetes.io/projected/97eb915b-40dc-4ab9-bde8-3410417ca78f-kube-api-access-2w2bp\") pod \"97eb915b-40dc-4ab9-bde8-3410417ca78f\" (UID: \"97eb915b-40dc-4ab9-bde8-3410417ca78f\") " Oct 30 16:43:48.550377 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:48.550346 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-openshift-ca" pod "97eb915b-40dc-4ab9-bde8-3410417ca78f" (UID: "97eb915b-40dc-4ab9-bde8-3410417ca78f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:48.552181 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:48.552152 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-xks-ca" pod "97eb915b-40dc-4ab9-bde8-3410417ca78f" (UID: "97eb915b-40dc-4ab9-bde8-3410417ca78f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:48.552305 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:48.552224 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97eb915b-40dc-4ab9-bde8-3410417ca78f-kube-api-access-2w2bp" pod "97eb915b-40dc-4ab9-bde8-3410417ca78f" (UID: "97eb915b-40dc-4ab9-bde8-3410417ca78f"). InnerVolumeSpecName "kube-api-access-2w2bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:48.651112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:48.651023 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:43:48.651112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:48.651056 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97eb915b-40dc-4ab9-bde8-3410417ca78f-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:43:48.651112 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:48.651067 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w2bp\" (UniqueName: \"kubernetes.io/projected/97eb915b-40dc-4ab9-bde8-3410417ca78f-kube-api-access-2w2bp\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:43:49.029172 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:49.029139 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd"] Oct 30 16:43:49.040645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:49.040609 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc5d5d7-cf9cd"] Oct 30 16:43:49.355677 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:49.355598 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" Oct 30 16:43:49.355677 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:49.355609 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j" event={"ID":"97eb915b-40dc-4ab9-bde8-3410417ca78f","Type":"ContainerDied","Data":"23e0008ac2a3c78884e43a592a31e4c7941d4f203b03008083fbdc704d0f1696"} Oct 30 16:43:49.355677 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:49.355654 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23e0008ac2a3c78884e43a592a31e4c7941d4f203b03008083fbdc704d0f1696" Oct 30 16:43:50.393987 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:50.393936 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bc4a05-8c71-4073-a2c3-779b80a3a3d9" path="/var/lib/kubelet/pods/f9bc4a05-8c71-4073-a2c3-779b80a3a3d9/volumes" Oct 30 16:43:51.028403 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:51.028368 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg"] Oct 30 16:43:51.039464 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:51.039430 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee7dd1a4-68pzg"] Oct 30 16:43:52.398823 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:52.398791 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555c9e26-cfa2-4413-81a5-f21a77718534" path="/var/lib/kubelet/pods/555c9e26-cfa2-4413-81a5-f21a77718534/volumes" Oct 30 16:43:55.930471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:55.930425 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz"] Oct 30 16:43:55.930867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:55.930839 5838 cpu_manager.go:414] "containerMap: removing container" podUID="97eb915b-40dc-4ab9-bde8-3410417ca78f" containerName="revoke-keys" Oct 30 16:43:55.930867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:55.930850 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97eb915b-40dc-4ab9-bde8-3410417ca78f" containerName="revoke-keys" Oct 30 16:43:55.930942 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:55.930919 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="97eb915b-40dc-4ab9-bde8-3410417ca78f" containerName="revoke-keys" Oct 30 16:43:55.935251 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:55.935233 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:55.961593 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:55.961564 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:43:55.961779 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:55.961626 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:43:55.964751 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:55.964726 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz"] Oct 30 16:43:56.024230 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.024191 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10928c79-040c-4191-b51b-797c5a496adc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jqldz\" (UID: \"10928c79-040c-4191-b51b-797c5a496adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:56.024364 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.024311 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10928c79-040c-4191-b51b-797c5a496adc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jqldz\" (UID: \"10928c79-040c-4191-b51b-797c5a496adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:56.024419 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.024391 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6s6n\" (UniqueName: \"kubernetes.io/projected/10928c79-040c-4191-b51b-797c5a496adc-kube-api-access-f6s6n\") pod \"maas-api-revoke-sub-3d4156f1-jqldz\" (UID: \"10928c79-040c-4191-b51b-797c5a496adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:56.042022 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.041991 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:43:56.042187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.042048 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:43:56.125351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.125315 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10928c79-040c-4191-b51b-797c5a496adc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jqldz\" (UID: \"10928c79-040c-4191-b51b-797c5a496adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:56.125520 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.125362 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6s6n\" (UniqueName: \"kubernetes.io/projected/10928c79-040c-4191-b51b-797c5a496adc-kube-api-access-f6s6n\") pod \"maas-api-revoke-sub-3d4156f1-jqldz\" (UID: \"10928c79-040c-4191-b51b-797c5a496adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:56.125520 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.125388 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10928c79-040c-4191-b51b-797c5a496adc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jqldz\" (UID: \"10928c79-040c-4191-b51b-797c5a496adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:56.126092 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.126069 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10928c79-040c-4191-b51b-797c5a496adc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jqldz\" (UID: \"10928c79-040c-4191-b51b-797c5a496adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:56.127966 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.127938 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10928c79-040c-4191-b51b-797c5a496adc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jqldz\" (UID: \"10928c79-040c-4191-b51b-797c5a496adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:56.148760 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.148725 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6s6n\" (UniqueName: \"kubernetes.io/projected/10928c79-040c-4191-b51b-797c5a496adc-kube-api-access-f6s6n\") pod \"maas-api-revoke-sub-3d4156f1-jqldz\" (UID: \"10928c79-040c-4191-b51b-797c5a496adc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:56.248474 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.248444 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:56.539435 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.539355 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:43:56.549466 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:56.549357 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz"] Oct 30 16:43:56.552473 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:43:56.552440 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10928c79_040c_4191_b51b_797c5a496adc.slice/crio-d251b6e34279733b4072806385d0446a3212d6b65dc9398dee8ce6683fd76691 WatchSource:0}: Error finding container d251b6e34279733b4072806385d0446a3212d6b65dc9398dee8ce6683fd76691: Status 404 returned error can't find the container with id d251b6e34279733b4072806385d0446a3212d6b65dc9398dee8ce6683fd76691 Oct 30 16:43:57.409543 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:57.409509 5838 generic.go:356] "Generic (PLEG): container finished" podID="10928c79-040c-4191-b51b-797c5a496adc" containerID="fc0ee3c75d8c727f6f3bf3ad68d120fbbff85fab113b2c2dce7eaaf3a6e476ce" exitCode=0 Oct 30 16:43:57.409943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:57.409588 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" event={"ID":"10928c79-040c-4191-b51b-797c5a496adc","Type":"ContainerDied","Data":"fc0ee3c75d8c727f6f3bf3ad68d120fbbff85fab113b2c2dce7eaaf3a6e476ce"} Oct 30 16:43:57.409943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:57.409642 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" event={"ID":"10928c79-040c-4191-b51b-797c5a496adc","Type":"ContainerStarted","Data":"d251b6e34279733b4072806385d0446a3212d6b65dc9398dee8ce6683fd76691"} Oct 30 16:43:58.550787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:58.550759 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:58.649059 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:58.649024 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10928c79-040c-4191-b51b-797c5a496adc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10928c79-040c-4191-b51b-797c5a496adc-maas-api-xks-ca\") pod \"10928c79-040c-4191-b51b-797c5a496adc\" (UID: \"10928c79-040c-4191-b51b-797c5a496adc\") " Oct 30 16:43:58.649227 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:58.649130 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10928c79-040c-4191-b51b-797c5a496adc-kube-api-access-f6s6n\" (UniqueName: \"kubernetes.io/projected/10928c79-040c-4191-b51b-797c5a496adc-kube-api-access-f6s6n\") pod \"10928c79-040c-4191-b51b-797c5a496adc\" (UID: \"10928c79-040c-4191-b51b-797c5a496adc\") " Oct 30 16:43:58.649227 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:58.649173 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10928c79-040c-4191-b51b-797c5a496adc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10928c79-040c-4191-b51b-797c5a496adc-maas-api-openshift-ca\") pod \"10928c79-040c-4191-b51b-797c5a496adc\" (UID: \"10928c79-040c-4191-b51b-797c5a496adc\") " Oct 30 16:43:58.649570 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:58.649533 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10928c79-040c-4191-b51b-797c5a496adc-maas-api-openshift-ca" pod "10928c79-040c-4191-b51b-797c5a496adc" (UID: "10928c79-040c-4191-b51b-797c5a496adc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:58.651295 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:58.651259 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10928c79-040c-4191-b51b-797c5a496adc-maas-api-xks-ca" pod "10928c79-040c-4191-b51b-797c5a496adc" (UID: "10928c79-040c-4191-b51b-797c5a496adc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:58.651406 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:58.651350 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10928c79-040c-4191-b51b-797c5a496adc-kube-api-access-f6s6n" pod "10928c79-040c-4191-b51b-797c5a496adc" (UID: "10928c79-040c-4191-b51b-797c5a496adc"). InnerVolumeSpecName "kube-api-access-f6s6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:58.750162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:58.750130 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6s6n\" (UniqueName: \"kubernetes.io/projected/10928c79-040c-4191-b51b-797c5a496adc-kube-api-access-f6s6n\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:43:58.750162 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:58.750161 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10928c79-040c-4191-b51b-797c5a496adc-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:43:58.750359 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:58.750173 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10928c79-040c-4191-b51b-797c5a496adc-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:43:59.424372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:59.424337 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" Oct 30 16:43:59.424555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:59.424371 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz" event={"ID":"10928c79-040c-4191-b51b-797c5a496adc","Type":"ContainerDied","Data":"d251b6e34279733b4072806385d0446a3212d6b65dc9398dee8ce6683fd76691"} Oct 30 16:43:59.424555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:43:59.424423 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d251b6e34279733b4072806385d0446a3212d6b65dc9398dee8ce6683fd76691" Oct 30 16:44:01.797333 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:44:01.797293 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:44:05.029508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:05.029470 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk"] Oct 30 16:44:05.040627 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:05.040597 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f3ff2b5-c9zdk"] Oct 30 16:44:06.395055 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:06.395022 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd6eb02-0ee3-4854-8b15-d391d7f74b66" path="/var/lib/kubelet/pods/5bd6eb02-0ee3-4854-8b15-d391d7f74b66/volumes" Oct 30 16:44:07.065607 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:07.065575 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp"] Oct 30 16:44:07.076444 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:07.076414 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57b0ebf-k7hvp"] Oct 30 16:44:07.079303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:07.079278 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:44:08.429994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:08.429201 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b89f1441-59fb-4d01-a5be-adf923324ff4" path="/var/lib/kubelet/pods/b89f1441-59fb-4d01-a5be-adf923324ff4/volumes" Oct 30 16:44:10.035994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:10.035265 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89"] Oct 30 16:44:10.062992 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:10.048599 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08677558-vmp89"] Oct 30 16:44:10.394593 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:10.394519 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac57fd23-359e-4db4-b110-2325b4e4960e" path="/var/lib/kubelet/pods/ac57fd23-359e-4db4-b110-2325b4e4960e/volumes" Oct 30 16:44:12.308606 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:12.308568 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:44:14.031695 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:14.031648 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr"] Oct 30 16:44:14.043025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:14.042990 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a87bebc-ntmxr"] Oct 30 16:44:14.395165 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:14.395080 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136e54ab-b8a1-46a4-8c4e-734d52222000" path="/var/lib/kubelet/pods/136e54ab-b8a1-46a4-8c4e-734d52222000/volumes" Oct 30 16:44:17.472267 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.472230 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz"] Oct 30 16:44:17.483879 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.483848 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jqldz"] Oct 30 16:44:17.525481 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.525447 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx"] Oct 30 16:44:17.525876 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.525863 5838 cpu_manager.go:414] "containerMap: removing container" podUID="10928c79-040c-4191-b51b-797c5a496adc" containerName="revoke-keys" Oct 30 16:44:17.525925 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.525877 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10928c79-040c-4191-b51b-797c5a496adc" containerName="revoke-keys" Oct 30 16:44:17.525959 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.525932 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="10928c79-040c-4191-b51b-797c5a496adc" containerName="revoke-keys" Oct 30 16:44:17.530342 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.530322 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:17.546592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.546557 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:44:17.546721 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.546700 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:44:17.549684 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.549656 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx"] Oct 30 16:44:17.550290 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.550271 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:44:17.550408 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.550347 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:44:17.559376 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.559349 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-642xp\" (UniqueName: \"kubernetes.io/projected/711072e8-136f-41d5-b300-80141d9f3c74-kube-api-access-642xp\") pod \"maas-api-revoke-sub-3d4156f1-6gnfx\" (UID: \"711072e8-136f-41d5-b300-80141d9f3c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:17.559520 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.559396 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gnfx\" (UID: \"711072e8-136f-41d5-b300-80141d9f3c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:17.559520 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.559496 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gnfx\" (UID: \"711072e8-136f-41d5-b300-80141d9f3c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:17.659891 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.659848 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gnfx\" (UID: \"711072e8-136f-41d5-b300-80141d9f3c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:17.660199 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.659924 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gnfx\" (UID: \"711072e8-136f-41d5-b300-80141d9f3c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:17.660199 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.660008 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-642xp\" (UniqueName: \"kubernetes.io/projected/711072e8-136f-41d5-b300-80141d9f3c74-kube-api-access-642xp\") pod \"maas-api-revoke-sub-3d4156f1-6gnfx\" (UID: \"711072e8-136f-41d5-b300-80141d9f3c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:17.660703 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.660680 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gnfx\" (UID: \"711072e8-136f-41d5-b300-80141d9f3c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:17.662269 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.662245 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gnfx\" (UID: \"711072e8-136f-41d5-b300-80141d9f3c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:17.683150 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.683126 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-642xp\" (UniqueName: \"kubernetes.io/projected/711072e8-136f-41d5-b300-80141d9f3c74-kube-api-access-642xp\") pod \"maas-api-revoke-sub-3d4156f1-6gnfx\" (UID: \"711072e8-136f-41d5-b300-80141d9f3c74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:17.843480 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:17.843442 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:18.149747 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:18.149674 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx"] Oct 30 16:44:18.153225 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:44:18.153189 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711072e8_136f_41d5_b300_80141d9f3c74.slice/crio-0db771d6743c0b0fc4b3c17c60636ff5853eadecedab358a3122ef858aec8593 WatchSource:0}: Error finding container 0db771d6743c0b0fc4b3c17c60636ff5853eadecedab358a3122ef858aec8593: Status 404 returned error can't find the container with id 0db771d6743c0b0fc4b3c17c60636ff5853eadecedab358a3122ef858aec8593 Oct 30 16:44:18.394806 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:18.394775 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10928c79-040c-4191-b51b-797c5a496adc" path="/var/lib/kubelet/pods/10928c79-040c-4191-b51b-797c5a496adc/volumes" Oct 30 16:44:18.571810 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:18.571768 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" event={"ID":"711072e8-136f-41d5-b300-80141d9f3c74","Type":"ContainerStarted","Data":"7bb783153962ffd4618c97f2b8a30cf5c790a98f814666a7a378dc1ec65058f4"} Oct 30 16:44:18.572561 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:18.571824 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" event={"ID":"711072e8-136f-41d5-b300-80141d9f3c74","Type":"ContainerStarted","Data":"0db771d6743c0b0fc4b3c17c60636ff5853eadecedab358a3122ef858aec8593"} Oct 30 16:44:18.623436 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:18.623387 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" podStartSLOduration=1.6233736300000001 podStartE2EDuration="1.62337363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:18.609808352 +0000 UTC m=+1420.358397739" watchObservedRunningTime="2026-10-30 16:44:18.62337363 +0000 UTC m=+1420.371963017" Oct 30 16:44:19.579446 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:19.579413 5838 generic.go:356] "Generic (PLEG): container finished" podID="711072e8-136f-41d5-b300-80141d9f3c74" containerID="7bb783153962ffd4618c97f2b8a30cf5c790a98f814666a7a378dc1ec65058f4" exitCode=0 Oct 30 16:44:19.579862 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:19.579497 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" event={"ID":"711072e8-136f-41d5-b300-80141d9f3c74","Type":"ContainerDied","Data":"7bb783153962ffd4618c97f2b8a30cf5c790a98f814666a7a378dc1ec65058f4"} Oct 30 16:44:20.723502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:20.723471 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:20.788424 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:20.788386 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-xks-ca\") pod \"711072e8-136f-41d5-b300-80141d9f3c74\" (UID: \"711072e8-136f-41d5-b300-80141d9f3c74\") " Oct 30 16:44:20.788582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:20.788493 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-openshift-ca\") pod \"711072e8-136f-41d5-b300-80141d9f3c74\" (UID: \"711072e8-136f-41d5-b300-80141d9f3c74\") " Oct 30 16:44:20.788624 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:20.788584 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711072e8-136f-41d5-b300-80141d9f3c74-kube-api-access-642xp\" (UniqueName: \"kubernetes.io/projected/711072e8-136f-41d5-b300-80141d9f3c74-kube-api-access-642xp\") pod \"711072e8-136f-41d5-b300-80141d9f3c74\" (UID: \"711072e8-136f-41d5-b300-80141d9f3c74\") " Oct 30 16:44:20.788865 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:20.788836 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-openshift-ca" pod "711072e8-136f-41d5-b300-80141d9f3c74" (UID: "711072e8-136f-41d5-b300-80141d9f3c74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:20.790671 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:20.790643 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-xks-ca" pod "711072e8-136f-41d5-b300-80141d9f3c74" (UID: "711072e8-136f-41d5-b300-80141d9f3c74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:44:20.790785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:20.790680 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711072e8-136f-41d5-b300-80141d9f3c74-kube-api-access-642xp" pod "711072e8-136f-41d5-b300-80141d9f3c74" (UID: "711072e8-136f-41d5-b300-80141d9f3c74"). InnerVolumeSpecName "kube-api-access-642xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:20.889380 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:20.889296 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-642xp\" (UniqueName: \"kubernetes.io/projected/711072e8-136f-41d5-b300-80141d9f3c74-kube-api-access-642xp\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:44:20.889380 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:20.889330 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:44:20.889380 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:20.889340 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711072e8-136f-41d5-b300-80141d9f3c74-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:44:21.599819 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:21.599788 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" Oct 30 16:44:21.599819 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:21.599794 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx" event={"ID":"711072e8-136f-41d5-b300-80141d9f3c74","Type":"ContainerDied","Data":"0db771d6743c0b0fc4b3c17c60636ff5853eadecedab358a3122ef858aec8593"} Oct 30 16:44:21.600048 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:21.599849 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0db771d6743c0b0fc4b3c17c60636ff5853eadecedab358a3122ef858aec8593" Oct 30 16:44:29.548003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:29.547942 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:44:30.057748 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:30.057695 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj"] Oct 30 16:44:30.072410 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:30.072375 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19cb1ac3-2s9nj"] Oct 30 16:44:30.394539 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:30.394455 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3546bb-7dcb-4642-bb39-411517d82ad3" path="/var/lib/kubelet/pods/fc3546bb-7dcb-4642-bb39-411517d82ad3/volumes" Oct 30 16:44:31.026726 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:31.026690 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z"] Oct 30 16:44:31.038102 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:31.038070 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25a490c3-cct6z"] Oct 30 16:44:31.796702 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:44:31.796663 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:44:32.399167 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:32.396743 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7940f372-7bea-44be-b44c-9a57aca6d584" path="/var/lib/kubelet/pods/7940f372-7bea-44be-b44c-9a57aca6d584/volumes" Oct 30 16:44:34.456370 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:34.456328 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:44:37.028479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:37.028443 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm"] Oct 30 16:44:37.046379 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:37.046334 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl"] Oct 30 16:44:37.048467 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:37.048434 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8697d674-cn9fm"] Oct 30 16:44:37.050561 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:37.050538 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad95481c-297xl"] Oct 30 16:44:38.395893 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:38.395860 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7a25b6-d5b4-470f-b5a6-66b57f2545d3" path="/var/lib/kubelet/pods/7d7a25b6-d5b4-470f-b5a6-66b57f2545d3/volumes" Oct 30 16:44:38.396346 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:38.396327 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ff126c2-5d26-47d2-8440-8d81421441de" path="/var/lib/kubelet/pods/9ff126c2-5d26-47d2-8440-8d81421441de/volumes" Oct 30 16:44:40.462187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:40.462108 5838 scope.go:122] "RemoveContainer" containerID="03bdba7b14e3781d8fd49d9cd6858c3121292ed7d4d6f4fe4049e8ae8a4dda7b" Oct 30 16:44:40.471901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:40.471881 5838 scope.go:122] "RemoveContainer" containerID="d5a8973969240e1e15a19fda55b42a518f687d748e210974cd5b9a2a57692fff" Oct 30 16:44:40.480598 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:40.480574 5838 scope.go:122] "RemoveContainer" containerID="94cf546ae3a4251684c08ae17af80bcb0171d18b84e8fa73508b986529721389" Oct 30 16:44:40.489213 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:40.489186 5838 scope.go:122] "RemoveContainer" containerID="f5e16d138ad281ea82f1a7b986fe90079605e318d460a45487caf9716b9a0991" Oct 30 16:44:40.497355 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:40.497336 5838 scope.go:122] "RemoveContainer" containerID="8a505c2b7eb38e091d5b59a61c2e5ab2b7f06e6b44c871623fb1a10404952416" Oct 30 16:44:40.505789 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:40.505770 5838 scope.go:122] "RemoveContainer" containerID="14941ad30a3852f02b01a98ef8f6878151035a0e089b7bc27f7d781f6d861ecc" Oct 30 16:44:40.513986 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:40.513950 5838 scope.go:122] "RemoveContainer" containerID="f32b928015922398650bb14eb66ca09ed2e8baf0bb16f40363e3b2324fe8b8aa" Oct 30 16:44:40.522492 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:40.522473 5838 scope.go:122] "RemoveContainer" containerID="e198e48d3d15bea096ed60df6822e10ff87319f1550788ba12c255f0acf92ebc" Oct 30 16:44:40.530884 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:40.530866 5838 scope.go:122] "RemoveContainer" containerID="d0a0139b58a1c802770a12bf2c5c89a6e014c0d01f9df0d5129e5f3f3d771de6" Oct 30 16:44:41.182555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.182518 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx"] Oct 30 16:44:41.193409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.193346 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gnfx"] Oct 30 16:44:41.233263 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.233225 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr"] Oct 30 16:44:41.233680 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.233662 5838 cpu_manager.go:414] "containerMap: removing container" podUID="711072e8-136f-41d5-b300-80141d9f3c74" containerName="revoke-keys" Oct 30 16:44:41.233729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.233683 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711072e8-136f-41d5-b300-80141d9f3c74" containerName="revoke-keys" Oct 30 16:44:41.233765 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.233752 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="711072e8-136f-41d5-b300-80141d9f3c74" containerName="revoke-keys" Oct 30 16:44:41.239393 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.239375 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:41.253907 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.253876 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:44:41.253907 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.253889 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:44:41.253907 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.253878 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:44:41.254111 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.253909 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:44:41.256048 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.256025 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr"] Oct 30 16:44:41.388805 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.388762 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jxkrr\" (UID: \"372ff1e4-0bac-457a-ab66-f174fd034767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:41.389049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.388943 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8fbf\" (UniqueName: \"kubernetes.io/projected/372ff1e4-0bac-457a-ab66-f174fd034767-kube-api-access-l8fbf\") pod \"maas-api-revoke-sub-3d4156f1-jxkrr\" (UID: \"372ff1e4-0bac-457a-ab66-f174fd034767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:41.389172 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.389155 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jxkrr\" (UID: \"372ff1e4-0bac-457a-ab66-f174fd034767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:41.489718 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.489682 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jxkrr\" (UID: \"372ff1e4-0bac-457a-ab66-f174fd034767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:41.490235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.489765 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8fbf\" (UniqueName: \"kubernetes.io/projected/372ff1e4-0bac-457a-ab66-f174fd034767-kube-api-access-l8fbf\") pod \"maas-api-revoke-sub-3d4156f1-jxkrr\" (UID: \"372ff1e4-0bac-457a-ab66-f174fd034767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:41.490235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.489787 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jxkrr\" (UID: \"372ff1e4-0bac-457a-ab66-f174fd034767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:41.490440 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.490418 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jxkrr\" (UID: \"372ff1e4-0bac-457a-ab66-f174fd034767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:41.492138 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.492113 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jxkrr\" (UID: \"372ff1e4-0bac-457a-ab66-f174fd034767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:41.512725 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.512697 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8fbf\" (UniqueName: \"kubernetes.io/projected/372ff1e4-0bac-457a-ab66-f174fd034767-kube-api-access-l8fbf\") pod \"maas-api-revoke-sub-3d4156f1-jxkrr\" (UID: \"372ff1e4-0bac-457a-ab66-f174fd034767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:41.552688 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.552662 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:41.855001 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:41.854871 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr"] Oct 30 16:44:41.859494 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:44:41.859461 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372ff1e4_0bac_457a_ab66_f174fd034767.slice/crio-245bc9c8f7c8519513391ce100112c9580862b95ab9dca37c41e4da60a09a062 WatchSource:0}: Error finding container 245bc9c8f7c8519513391ce100112c9580862b95ab9dca37c41e4da60a09a062: Status 404 returned error can't find the container with id 245bc9c8f7c8519513391ce100112c9580862b95ab9dca37c41e4da60a09a062 Oct 30 16:44:42.394878 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:42.394800 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711072e8-136f-41d5-b300-80141d9f3c74" path="/var/lib/kubelet/pods/711072e8-136f-41d5-b300-80141d9f3c74/volumes" Oct 30 16:44:42.745354 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:42.745321 5838 generic.go:356] "Generic (PLEG): container finished" podID="372ff1e4-0bac-457a-ab66-f174fd034767" containerID="8a97d7a9d5e0645112fec15916d9a3b90a3d3edbe1ba7d570ce41dd525c58586" exitCode=0 Oct 30 16:44:42.745792 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:42.745415 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" event={"ID":"372ff1e4-0bac-457a-ab66-f174fd034767","Type":"ContainerDied","Data":"8a97d7a9d5e0645112fec15916d9a3b90a3d3edbe1ba7d570ce41dd525c58586"} Oct 30 16:44:42.745792 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:42.745483 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" event={"ID":"372ff1e4-0bac-457a-ab66-f174fd034767","Type":"ContainerStarted","Data":"245bc9c8f7c8519513391ce100112c9580862b95ab9dca37c41e4da60a09a062"} Oct 30 16:44:43.890314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:43.890283 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:44.014016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:44.013910 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-xks-ca\") pod \"372ff1e4-0bac-457a-ab66-f174fd034767\" (UID: \"372ff1e4-0bac-457a-ab66-f174fd034767\") " Oct 30 16:44:44.014200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:44.014038 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-openshift-ca\") pod \"372ff1e4-0bac-457a-ab66-f174fd034767\" (UID: \"372ff1e4-0bac-457a-ab66-f174fd034767\") " Oct 30 16:44:44.014200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:44.014105 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372ff1e4-0bac-457a-ab66-f174fd034767-kube-api-access-l8fbf\" (UniqueName: \"kubernetes.io/projected/372ff1e4-0bac-457a-ab66-f174fd034767-kube-api-access-l8fbf\") pod \"372ff1e4-0bac-457a-ab66-f174fd034767\" (UID: \"372ff1e4-0bac-457a-ab66-f174fd034767\") " Oct 30 16:44:44.014397 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:44.014373 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-openshift-ca" pod "372ff1e4-0bac-457a-ab66-f174fd034767" (UID: "372ff1e4-0bac-457a-ab66-f174fd034767"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:44.016105 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:44.016083 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-xks-ca" pod "372ff1e4-0bac-457a-ab66-f174fd034767" (UID: "372ff1e4-0bac-457a-ab66-f174fd034767"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:44:44.016187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:44.016145 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372ff1e4-0bac-457a-ab66-f174fd034767-kube-api-access-l8fbf" pod "372ff1e4-0bac-457a-ab66-f174fd034767" (UID: "372ff1e4-0bac-457a-ab66-f174fd034767"). InnerVolumeSpecName "kube-api-access-l8fbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:44.115558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:44.115517 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8fbf\" (UniqueName: \"kubernetes.io/projected/372ff1e4-0bac-457a-ab66-f174fd034767-kube-api-access-l8fbf\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:44:44.115558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:44.115552 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:44:44.115558 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:44.115562 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372ff1e4-0bac-457a-ab66-f174fd034767-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:44:44.759438 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:44.759399 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" event={"ID":"372ff1e4-0bac-457a-ab66-f174fd034767","Type":"ContainerDied","Data":"245bc9c8f7c8519513391ce100112c9580862b95ab9dca37c41e4da60a09a062"} Oct 30 16:44:44.759589 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:44.759445 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr" Oct 30 16:44:44.759589 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:44.759461 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="245bc9c8f7c8519513391ce100112c9580862b95ab9dca37c41e4da60a09a062" Oct 30 16:44:53.198823 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:53.198742 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:44:58.052133 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:44:58.052097 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:45:00.166719 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.166675 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm"] Oct 30 16:45:00.167315 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.167296 5838 cpu_manager.go:414] "containerMap: removing container" podUID="372ff1e4-0bac-457a-ab66-f174fd034767" containerName="revoke-keys" Oct 30 16:45:00.167361 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.167317 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372ff1e4-0bac-457a-ab66-f174fd034767" containerName="revoke-keys" Oct 30 16:45:00.167449 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.167438 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="372ff1e4-0bac-457a-ab66-f174fd034767" containerName="revoke-keys" Oct 30 16:45:00.171017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.170991 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" Oct 30 16:45:00.173833 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.173810 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpnrl\" (UniqueName: \"kubernetes.io/projected/de62b9bf-270e-45c5-9444-0bb9fe1875be-kube-api-access-tpnrl\") pod \"maas-api-key-cleanup-29889645-9rkvm\" (UID: \"de62b9bf-270e-45c5-9444-0bb9fe1875be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" Oct 30 16:45:00.173954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.173868 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de62b9bf-270e-45c5-9444-0bb9fe1875be-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889645-9rkvm\" (UID: \"de62b9bf-270e-45c5-9444-0bb9fe1875be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" Oct 30 16:45:00.191463 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.191430 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm"] Oct 30 16:45:00.200743 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.200709 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:45:00.200743 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.200721 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:45:00.200955 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.200786 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:45:00.274696 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.274654 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpnrl\" (UniqueName: \"kubernetes.io/projected/de62b9bf-270e-45c5-9444-0bb9fe1875be-kube-api-access-tpnrl\") pod \"maas-api-key-cleanup-29889645-9rkvm\" (UID: \"de62b9bf-270e-45c5-9444-0bb9fe1875be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" Oct 30 16:45:00.274909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.274747 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de62b9bf-270e-45c5-9444-0bb9fe1875be-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889645-9rkvm\" (UID: \"de62b9bf-270e-45c5-9444-0bb9fe1875be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" Oct 30 16:45:00.275464 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.275435 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de62b9bf-270e-45c5-9444-0bb9fe1875be-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889645-9rkvm\" (UID: \"de62b9bf-270e-45c5-9444-0bb9fe1875be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" Oct 30 16:45:00.297946 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.297917 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpnrl\" (UniqueName: \"kubernetes.io/projected/de62b9bf-270e-45c5-9444-0bb9fe1875be-kube-api-access-tpnrl\") pod \"maas-api-key-cleanup-29889645-9rkvm\" (UID: \"de62b9bf-270e-45c5-9444-0bb9fe1875be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" Oct 30 16:45:00.486007 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.485954 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" Oct 30 16:45:00.786554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.786476 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm"] Oct 30 16:45:00.790271 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:45:00.790242 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde62b9bf_270e_45c5_9444_0bb9fe1875be.slice/crio-23b7414c0757f54c8b0d2063730151d01c64df76b9767979803aedbec02a60fd WatchSource:0}: Error finding container 23b7414c0757f54c8b0d2063730151d01c64df76b9767979803aedbec02a60fd: Status 404 returned error can't find the container with id 23b7414c0757f54c8b0d2063730151d01c64df76b9767979803aedbec02a60fd Oct 30 16:45:00.860744 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:00.860708 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" event={"ID":"de62b9bf-270e-45c5-9444-0bb9fe1875be","Type":"ContainerStarted","Data":"23b7414c0757f54c8b0d2063730151d01c64df76b9767979803aedbec02a60fd"} Oct 30 16:45:01.796424 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:45:01.796387 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:45:01.867956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:01.867854 5838 generic.go:356] "Generic (PLEG): container finished" podID="de62b9bf-270e-45c5-9444-0bb9fe1875be" containerID="86b83e21e6c18d2e614d52a24c90b85c70b228927b957861df9066a49561f976" exitCode=60 Oct 30 16:45:01.867956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:01.867935 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" event={"ID":"de62b9bf-270e-45c5-9444-0bb9fe1875be","Type":"ContainerDied","Data":"86b83e21e6c18d2e614d52a24c90b85c70b228927b957861df9066a49561f976"} Oct 30 16:45:01.868194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:01.868180 5838 scope.go:122] "RemoveContainer" containerID="86b83e21e6c18d2e614d52a24c90b85c70b228927b957861df9066a49561f976" Oct 30 16:45:02.876273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:02.876185 5838 generic.go:356] "Generic (PLEG): container finished" podID="de62b9bf-270e-45c5-9444-0bb9fe1875be" containerID="dabc00f62215cc2665e863ce0a4326923d7d0dad17292ca063488a0af4e9786d" exitCode=60 Oct 30 16:45:02.876273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:02.876258 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" event={"ID":"de62b9bf-270e-45c5-9444-0bb9fe1875be","Type":"ContainerDied","Data":"dabc00f62215cc2665e863ce0a4326923d7d0dad17292ca063488a0af4e9786d"} Oct 30 16:45:02.876808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:02.876344 5838 scope.go:122] "RemoveContainer" containerID="86b83e21e6c18d2e614d52a24c90b85c70b228927b957861df9066a49561f976" Oct 30 16:45:02.876808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:02.876644 5838 scope.go:122] "RemoveContainer" containerID="dabc00f62215cc2665e863ce0a4326923d7d0dad17292ca063488a0af4e9786d" Oct 30 16:45:02.877025 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:45:02.876959 5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889645-9rkvm_odh-ai-gateway-infra(de62b9bf-270e-45c5-9444-0bb9fe1875be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" Oct 30 16:45:03.061987 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.061943 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826"] Oct 30 16:45:03.068109 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.068069 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.082743 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.082712 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-3bc669-openshift-default-dockercfg-885k5\"" Oct 30 16:45:03.092943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.092899 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826"] Oct 30 16:45:03.099567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.099534 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-data\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.099732 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.099606 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99cm2\" (UniqueName: \"kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-kube-api-access-99cm2\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.099732 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.099655 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91179b4c-4588-46ac-af09-924ed33cac64-istio-podinfo\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.099865 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.099741 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91179b4c-4588-46ac-af09-924ed33cac64-istiod-ca-cert\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.099865 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.099789 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-istio-token\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.099865 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.099820 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-envoy\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.100026 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.099957 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-socket\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.100099 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.100060 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-credential-socket\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.100274 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.100153 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-certs\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.200679 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.200589 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-certs\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.200679 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.200652 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-data\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.200903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.200696 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99cm2\" (UniqueName: \"kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-kube-api-access-99cm2\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.200903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.200734 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91179b4c-4588-46ac-af09-924ed33cac64-istio-podinfo\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.200903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.200762 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91179b4c-4588-46ac-af09-924ed33cac64-istiod-ca-cert\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.200903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.200786 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-istio-token\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.200903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.200811 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-envoy\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.200903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.200856 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-socket\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.200903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.200888 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-credential-socket\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.201327 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.201153 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-certs\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.201327 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.201223 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-data\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.201327 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.201242 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-credential-socket\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.201497 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.201422 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-socket\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.201650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.201628 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91179b4c-4588-46ac-af09-924ed33cac64-istiod-ca-cert\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.203387 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.203356 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-envoy\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.203534 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.203514 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91179b4c-4588-46ac-af09-924ed33cac64-istio-podinfo\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.223441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.223415 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-istio-token\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.223630 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.223608 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99cm2\" (UniqueName: \"kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-kube-api-access-99cm2\") pod \"e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.383154 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.383124 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:03.707349 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.707315 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826"] Oct 30 16:45:03.710547 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:45:03.710518 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91179b4c_4588_46ac_af09_924ed33cac64.slice/crio-07646e1e5a82f38025d2936108420dd3a2e56b85e5a73b90e084f6ebdf25cc11 WatchSource:0}: Error finding container 07646e1e5a82f38025d2936108420dd3a2e56b85e5a73b90e084f6ebdf25cc11: Status 404 returned error can't find the container with id 07646e1e5a82f38025d2936108420dd3a2e56b85e5a73b90e084f6ebdf25cc11 Oct 30 16:45:03.712755 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.712723 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:45:03.712842 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.712790 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:45:03.712842 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.712818 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:45:03.885172 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.885126 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" event={"ID":"91179b4c-4588-46ac-af09-924ed33cac64","Type":"ContainerStarted","Data":"8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34"} Oct 30 16:45:03.885625 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.885192 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" event={"ID":"91179b4c-4588-46ac-af09-924ed33cac64","Type":"ContainerStarted","Data":"07646e1e5a82f38025d2936108420dd3a2e56b85e5a73b90e084f6ebdf25cc11"} Oct 30 16:45:03.887240 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.887220 5838 scope.go:122] "RemoveContainer" containerID="dabc00f62215cc2665e863ce0a4326923d7d0dad17292ca063488a0af4e9786d" Oct 30 16:45:03.887510 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:45:03.887487 5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889645-9rkvm_odh-ai-gateway-infra(de62b9bf-270e-45c5-9444-0bb9fe1875be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" Oct 30 16:45:03.929782 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:03.929728 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" podStartSLOduration=0.929714123 podStartE2EDuration="929.714123ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.928691621 +0000 UTC m=+1465.677281008" watchObservedRunningTime="2026-10-30 16:45:03.929714123 +0000 UTC m=+1465.678303510" Oct 30 16:45:04.384281 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:04.384247 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:04.394561 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:04.394532 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:04.893783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:04.893755 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:04.894847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:04.894826 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:07.929832 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:07.929794 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5"] Oct 30 16:45:07.933836 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:07.933811 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:07.959353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:07.959321 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3bc669-serving-cert\"" Oct 30 16:45:07.959518 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:07.959413 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 16:45:07.959518 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:07.959432 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dqwfr\"" Oct 30 16:45:07.964539 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:07.964509 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5"] Oct 30 16:45:08.048907 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.048866 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7rt\" (UniqueName: \"kubernetes.io/projected/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-kube-api-access-qp7rt\") pod \"maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.049115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.049052 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.049115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.049085 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.049115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.049108 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.150257 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.150212 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7rt\" (UniqueName: \"kubernetes.io/projected/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-kube-api-access-qp7rt\") pod \"maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.150457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.150297 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.150457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.150323 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.150457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.150347 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.151119 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.151091 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.153071 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.153045 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.153174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.153096 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.179952 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.179883 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7rt\" (UniqueName: \"kubernetes.io/projected/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-kube-api-access-qp7rt\") pod \"maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.199866 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.199836 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv"] Oct 30 16:45:08.203834 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.203815 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" Oct 30 16:45:08.223061 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.223021 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv"] Oct 30 16:45:08.224362 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.224339 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-42c81718-dockercfg-4b9nb\"" Oct 30 16:45:08.247536 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.247502 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:08.251832 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.251802 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml82d\" (UniqueName: \"kubernetes.io/projected/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-kube-api-access-ml82d\") pod \"payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv\" (UID: \"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" Oct 30 16:45:08.251958 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.251857 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv\" (UID: \"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" Oct 30 16:45:08.277280 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.277254 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3bc669\"" Oct 30 16:45:08.368082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.367014 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml82d\" (UniqueName: \"kubernetes.io/projected/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-kube-api-access-ml82d\") pod \"payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv\" (UID: \"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" Oct 30 16:45:08.368082 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.367108 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv\" (UID: \"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" Oct 30 16:45:08.368802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.368457 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv\" (UID: \"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" Oct 30 16:45:08.393114 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.393063 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml82d\" (UniqueName: \"kubernetes.io/projected/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-kube-api-access-ml82d\") pod \"payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv\" (UID: \"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" Oct 30 16:45:08.518350 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.518319 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" Oct 30 16:45:08.607370 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.607338 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5"] Oct 30 16:45:08.609280 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:45:08.609246 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd043f3c_4e08_4d7c_9c53_2260f8cf5582.slice/crio-5eaa19f93e6e82c5d7b65c1ed1373dc44aa42ebe9cd94d9bf0e0b1d0b0af9d7c WatchSource:0}: Error finding container 5eaa19f93e6e82c5d7b65c1ed1373dc44aa42ebe9cd94d9bf0e0b1d0b0af9d7c: Status 404 returned error can't find the container with id 5eaa19f93e6e82c5d7b65c1ed1373dc44aa42ebe9cd94d9bf0e0b1d0b0af9d7c Oct 30 16:45:08.610817 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.610798 5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:45:08.821351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.821274 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv"] Oct 30 16:45:08.824806 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:45:08.824766 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa30de4_1e7a_4cf1_8b88_c6f2d66df65f.slice/crio-20c849f7f695d529cdc1be923b5647c31cd883b03adab53a5cf5a638dca00ff8 WatchSource:0}: Error finding container 20c849f7f695d529cdc1be923b5647c31cd883b03adab53a5cf5a638dca00ff8: Status 404 returned error can't find the container with id 20c849f7f695d529cdc1be923b5647c31cd883b03adab53a5cf5a638dca00ff8 Oct 30 16:45:08.921444 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.921405 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" event={"ID":"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f","Type":"ContainerStarted","Data":"08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6"} Oct 30 16:45:08.921649 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.921481 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" Oct 30 16:45:08.921649 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.921511 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" event={"ID":"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f","Type":"ContainerStarted","Data":"20c849f7f695d529cdc1be923b5647c31cd883b03adab53a5cf5a638dca00ff8"} Oct 30 16:45:08.922886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.922860 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" event={"ID":"dd043f3c-4e08-4d7c-9c53-2260f8cf5582","Type":"ContainerStarted","Data":"5eaa19f93e6e82c5d7b65c1ed1373dc44aa42ebe9cd94d9bf0e0b1d0b0af9d7c"} Oct 30 16:45:08.959711 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:08.959648 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" podStartSLOduration=0.959631175 podStartE2EDuration="959.631175ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:08.959268929 +0000 UTC m=+1470.707858330" watchObservedRunningTime="2026-10-30 16:45:08.959631175 +0000 UTC m=+1470.708220564" Oct 30 16:45:09.932028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:09.931957 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" event={"ID":"dd043f3c-4e08-4d7c-9c53-2260f8cf5582","Type":"ContainerStarted","Data":"f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35"} Oct 30 16:45:09.932226 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:09.932042 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:09.976910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:09.976852 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" podStartSLOduration=2.552795012 podStartE2EDuration="2.976837545s" totalImagesPullingTime="424.042533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:09.975595417 +0000 UTC m=+1471.724184834" watchObservedRunningTime="2026-10-30 16:45:09.976837545 +0000 UTC m=+1471.725426931" Oct 30 16:45:11.722935 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:11.722889 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:45:15.944664 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:15.944635 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:45:16.388356 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:16.388322 5838 scope.go:122] "RemoveContainer" containerID="dabc00f62215cc2665e863ce0a4326923d7d0dad17292ca063488a0af4e9786d" Oct 30 16:45:16.983000 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:16.982889 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" event={"ID":"de62b9bf-270e-45c5-9444-0bb9fe1875be","Type":"ContainerStarted","Data":"544f7a9e847e6562f9574f18283e01fce82136a94faa122b6634b62d0c56f0e4"} Oct 30 16:45:17.991174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:17.991138 5838 generic.go:356] "Generic (PLEG): container finished" podID="de62b9bf-270e-45c5-9444-0bb9fe1875be" containerID="544f7a9e847e6562f9574f18283e01fce82136a94faa122b6634b62d0c56f0e4" exitCode=60 Oct 30 16:45:17.991613 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:17.991211 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" event={"ID":"de62b9bf-270e-45c5-9444-0bb9fe1875be","Type":"ContainerDied","Data":"544f7a9e847e6562f9574f18283e01fce82136a94faa122b6634b62d0c56f0e4"} Oct 30 16:45:17.991613 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:17.991269 5838 scope.go:122] "RemoveContainer" containerID="dabc00f62215cc2665e863ce0a4326923d7d0dad17292ca063488a0af4e9786d" Oct 30 16:45:17.991613 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:17.991590 5838 scope.go:122] "RemoveContainer" containerID="544f7a9e847e6562f9574f18283e01fce82136a94faa122b6634b62d0c56f0e4" Oct 30 16:45:17.991867 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:45:17.991845 5838 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889645-9rkvm_odh-ai-gateway-infra(de62b9bf-270e-45c5-9444-0bb9fe1875be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" Oct 30 16:45:18.038541 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:18.038469 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" podStartSLOduration=18.038446498 podStartE2EDuration="18.038446498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:17.037010575 +0000 UTC m=+1478.785599964" watchObservedRunningTime="2026-10-30 16:45:18.038446498 +0000 UTC m=+1479.787035886" Oct 30 16:45:18.039956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:18.039922 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm"] Oct 30 16:45:19.172510 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:19.172481 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" Oct 30 16:45:19.318589 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:19.318501 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de62b9bf-270e-45c5-9444-0bb9fe1875be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de62b9bf-270e-45c5-9444-0bb9fe1875be-maas-api-service-ca\") pod \"de62b9bf-270e-45c5-9444-0bb9fe1875be\" (UID: \"de62b9bf-270e-45c5-9444-0bb9fe1875be\") " Oct 30 16:45:19.318589 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:19.318570 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de62b9bf-270e-45c5-9444-0bb9fe1875be-kube-api-access-tpnrl\" (UniqueName: \"kubernetes.io/projected/de62b9bf-270e-45c5-9444-0bb9fe1875be-kube-api-access-tpnrl\") pod \"de62b9bf-270e-45c5-9444-0bb9fe1875be\" (UID: \"de62b9bf-270e-45c5-9444-0bb9fe1875be\") " Oct 30 16:45:19.318900 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:19.318877 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de62b9bf-270e-45c5-9444-0bb9fe1875be-maas-api-service-ca" pod "de62b9bf-270e-45c5-9444-0bb9fe1875be" (UID: "de62b9bf-270e-45c5-9444-0bb9fe1875be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:19.320783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:19.320756 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de62b9bf-270e-45c5-9444-0bb9fe1875be-kube-api-access-tpnrl" pod "de62b9bf-270e-45c5-9444-0bb9fe1875be" (UID: "de62b9bf-270e-45c5-9444-0bb9fe1875be"). InnerVolumeSpecName "kube-api-access-tpnrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:19.419554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:19.419519 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de62b9bf-270e-45c5-9444-0bb9fe1875be-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:45:19.419554 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:19.419548 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpnrl\" (UniqueName: \"kubernetes.io/projected/de62b9bf-270e-45c5-9444-0bb9fe1875be-kube-api-access-tpnrl\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:45:19.935786 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:19.935754 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" Oct 30 16:45:20.010877 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:20.010838 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" event={"ID":"de62b9bf-270e-45c5-9444-0bb9fe1875be","Type":"ContainerDied","Data":"23b7414c0757f54c8b0d2063730151d01c64df76b9767979803aedbec02a60fd"} Oct 30 16:45:20.010877 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:20.010860 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm" Oct 30 16:45:20.011130 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:20.010902 5838 scope.go:122] "RemoveContainer" containerID="544f7a9e847e6562f9574f18283e01fce82136a94faa122b6634b62d0c56f0e4" Oct 30 16:45:20.063840 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:20.063807 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm"] Oct 30 16:45:20.075716 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:20.075665 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-9rkvm"] Oct 30 16:45:20.403163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:20.403122 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" path="/var/lib/kubelet/pods/de62b9bf-270e-45c5-9444-0bb9fe1875be/volumes" Oct 30 16:45:20.935180 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:20.935133 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:45:20.935351 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:20.935210 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:22.027027 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:22.026992 5838 generic.go:356] "Generic (PLEG): container finished" podID="91179b4c-4588-46ac-af09-924ed33cac64" containerID="8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34" exitCode=0 Oct 30 16:45:22.027592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:22.027082 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" event={"ID":"91179b4c-4588-46ac-af09-924ed33cac64","Type":"ContainerDied","Data":"8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34"} Oct 30 16:45:22.027670 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:22.027596 5838 scope.go:122] "RemoveContainer" containerID="8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34" Oct 30 16:45:22.028849 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:22.028823 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:45:22.028996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:22.028890 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:45:22.028996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:22.028920 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:45:23.044041 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.044001 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" event={"ID":"91179b4c-4588-46ac-af09-924ed33cac64","Type":"ContainerStarted","Data":"2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59"} Oct 30 16:45:23.287084 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.287044 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d"] Oct 30 16:45:23.287508 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.287491 5838 cpu_manager.go:414] "containerMap: removing container" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" containerName="cleanup" Oct 30 16:45:23.287591 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.287509 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" containerName="cleanup" Oct 30 16:45:23.287591 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.287537 5838 cpu_manager.go:414] "containerMap: removing container" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" containerName="cleanup" Oct 30 16:45:23.287591 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.287542 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" containerName="cleanup" Oct 30 16:45:23.287700 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.287613 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" containerName="cleanup" Oct 30 16:45:23.287700 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.287626 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" containerName="cleanup" Oct 30 16:45:23.290745 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.290721 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.306569 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.306541 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3bc669\"/\"default-dockercfg-kzmpl\"" Oct 30 16:45:23.306709 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.306541 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3bc669\"/\"kube-root-ca.crt\"" Oct 30 16:45:23.306709 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.306578 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3bc669\"/\"simulator-single-main-3bc669-kserve-self-signed-certs\"" Oct 30 16:45:23.306709 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.306543 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3bc669\"/\"openshift-service-ca.crt\"" Oct 30 16:45:23.340993 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.339958 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d"] Oct 30 16:45:23.364778 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.364744 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7891ff33-b413-4506-81cb-bc99d3a20c8e-tls-certs\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.364778 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.364781 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-dshm\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.365009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.364801 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-model-cache\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.365009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.364932 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-tmp-dir\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.365009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.365001 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm9ml\" (UniqueName: \"kubernetes.io/projected/7891ff33-b413-4506-81cb-bc99d3a20c8e-kube-api-access-wm9ml\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.365122 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.365077 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-home\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.383934 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.383905 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:23.384137 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.384112 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 16:45:23.384183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.384169 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 16:45:23.466034 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.465994 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-home\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.466206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.466051 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7891ff33-b413-4506-81cb-bc99d3a20c8e-tls-certs\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.466206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.466168 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-dshm\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.466300 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.466207 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-model-cache\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.466360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.466312 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-tmp-dir\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.466415 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.466378 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm9ml\" (UniqueName: \"kubernetes.io/projected/7891ff33-b413-4506-81cb-bc99d3a20c8e-kube-api-access-wm9ml\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.466473 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.466436 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-home\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.466666 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.466644 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-model-cache\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.466809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.466788 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-tmp-dir\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.468524 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.468505 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-dshm\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.468698 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.468681 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7891ff33-b413-4506-81cb-bc99d3a20c8e-tls-certs\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.489595 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.489568 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm9ml\" (UniqueName: \"kubernetes.io/projected/7891ff33-b413-4506-81cb-bc99d3a20c8e-kube-api-access-wm9ml\") pod \"simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.602874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.602787 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:23.941445 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:23.941418 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d"] Oct 30 16:45:23.943227 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:45:23.943199 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7891ff33_b413_4506_81cb_bc99d3a20c8e.slice/crio-2ca495bbad05a0cb25e7730bcf057e453df30e3a4402da421f38e0a4842f10b6 WatchSource:0}: Error finding container 2ca495bbad05a0cb25e7730bcf057e453df30e3a4402da421f38e0a4842f10b6: Status 404 returned error can't find the container with id 2ca495bbad05a0cb25e7730bcf057e453df30e3a4402da421f38e0a4842f10b6 Oct 30 16:45:24.058417 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:24.058384 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" event={"ID":"7891ff33-b413-4506-81cb-bc99d3a20c8e","Type":"ContainerStarted","Data":"80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1"} Oct 30 16:45:24.058820 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:24.058802 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" event={"ID":"7891ff33-b413-4506-81cb-bc99d3a20c8e","Type":"ContainerStarted","Data":"2ca495bbad05a0cb25e7730bcf057e453df30e3a4402da421f38e0a4842f10b6"} Oct 30 16:45:24.102285 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:24.102233 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" podStartSLOduration=1.102218029 podStartE2EDuration="1.102218029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:24.099077952 +0000 UTC m=+1485.847667360" watchObservedRunningTime="2026-10-30 16:45:24.102218029 +0000 UTC m=+1485.850807412" Oct 30 16:45:24.383894 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:24.383848 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 16:45:24.384102 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:24.383934 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 16:45:24.603718 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:24.603667 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:25.383997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:25.383944 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 16:45:25.384408 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:25.384013 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 16:45:25.644228 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:25.644150 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:45:26.384261 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:26.384222 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 16:45:26.384757 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:26.384287 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 16:45:27.042955 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:27.042925 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl"] Oct 30 16:45:27.054023 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:27.053994 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-zjwvl"] Oct 30 16:45:27.384212 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:27.383996 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 16:45:27.384212 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:27.384061 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 16:45:28.396224 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:28.396188 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ace71c4-cb6f-4835-8223-7832e17454f3" path="/var/lib/kubelet/pods/7ace71c4-cb6f-4835-8223-7832e17454f3/volumes" Oct 30 16:45:28.664955 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:28.664867 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:28.665372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:28.665338 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:28.666441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:28.666418 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:45:31.797651 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:45:31.797605 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:45:33.603724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:33.603688 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:33.616670 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:33.616642 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:34.140729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:34.140694 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:45:38.624663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:38.624621 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:45:38.631429 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:38.631406 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:45:40.597049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.597017 5838 scope.go:122] "RemoveContainer" containerID="3eaca676701827609d03dfda05e808b2358a91e7a95645c8f28d18f47baba94b" Oct 30 16:45:40.606548 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.606523 5838 scope.go:122] "RemoveContainer" containerID="91f3c80f562d51e77d019b0c138be134d7dd2cec37f7105887c712b4de379fe0" Oct 30 16:45:40.615420 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.615398 5838 scope.go:122] "RemoveContainer" containerID="a91344d841478ea418fa68ea59da417c9d7ca55dd23b0bc8f3e9618eee81c2dd" Oct 30 16:45:40.624037 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.624017 5838 scope.go:122] "RemoveContainer" containerID="a7092a95f28cd68277f9e06d670476102f78fea94aad183de06c993ee7c83c81" Oct 30 16:45:40.632098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.632079 5838 scope.go:122] "RemoveContainer" containerID="89c9209ee2b8d14d0aa8e20f2325eba570a308ed8b1602829c80f2d2a1b361d7" Oct 30 16:45:40.640465 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.640448 5838 scope.go:122] "RemoveContainer" containerID="e72d611265c2dd9bc2682ab9c2352cb3849c6b284dc5a79dbe671f44b115c484" Oct 30 16:45:40.648152 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.648131 5838 scope.go:122] "RemoveContainer" containerID="1caa717c263423c75e99c675b43bc367086791b0b629ca11ff10bfd61f59b555" Oct 30 16:45:40.658832 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.658809 5838 scope.go:122] "RemoveContainer" containerID="8358b356aba757cc66ddbc5d9cceafaf009386b1213a45c6ee0251ed6b736932" Oct 30 16:45:40.666765 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.666744 5838 scope.go:122] "RemoveContainer" containerID="dcade81ba514f0025d77737f283f3dae0accf131f2d107fae925b06163eb53dd" Oct 30 16:45:40.674347 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.674329 5838 scope.go:122] "RemoveContainer" containerID="2659d0dcebb744aa43882cb560ec9b2ac67eceb563a24959bba23f768683bc8a" Oct 30 16:45:40.682003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.681986 5838 scope.go:122] "RemoveContainer" containerID="51e245f5ea2b7226b34a90f8530e1170037bf2d6d7b225ecb0fd9511b165af9c" Oct 30 16:45:40.689945 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.689928 5838 scope.go:122] "RemoveContainer" containerID="11bf3aba9dce27d671983940ca7cb7affebc2d1ed64eea459154819d687aabd2" Oct 30 16:45:40.697635 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.697619 5838 scope.go:122] "RemoveContainer" containerID="8e6791fb634445b2a39cdf30841360e846c44a7542e1db2985024af297ef9a65" Oct 30 16:45:40.705606 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.705581 5838 scope.go:122] "RemoveContainer" containerID="823901dcf75c5dfc08eb189a542927860cf2abd2597ed4d17a87d9beadc9fa38" Oct 30 16:45:40.947743 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:40.947654 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:45:58.344962 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:45:58.344927 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:46:01.813613 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:46:01.811095 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:46:03.072298 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:03.072262 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d"] Oct 30 16:46:08.765409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:08.765367 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv"] Oct 30 16:46:11.790060 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:11.790024 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5"] Oct 30 16:46:11.790615 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:11.790266 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" podUID="dd043f3c-4e08-4d7c-9c53-2260f8cf5582" containerName="maas-api" containerID="cri-o://f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35" gracePeriod=30 Oct 30 16:46:12.267799 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.267770 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:46:12.373266 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.373182 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-maas-api-tls\") pod \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " Oct 30 16:46:12.373266 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.373233 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-openshift-service-ca\") pod \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " Oct 30 16:46:12.373506 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.373310 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-kube-api-access-qp7rt\" (UniqueName: \"kubernetes.io/projected/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-kube-api-access-qp7rt\") pod \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " Oct 30 16:46:12.373506 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.373362 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-metrics-certs\") pod \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\" (UID: \"dd043f3c-4e08-4d7c-9c53-2260f8cf5582\") " Oct 30 16:46:12.373682 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.373648 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-openshift-service-ca" pod "dd043f3c-4e08-4d7c-9c53-2260f8cf5582" (UID: "dd043f3c-4e08-4d7c-9c53-2260f8cf5582"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:12.373808 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.373759 5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-openshift-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:12.376261 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.376221 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-maas-api-tls" pod "dd043f3c-4e08-4d7c-9c53-2260f8cf5582" (UID: "dd043f3c-4e08-4d7c-9c53-2260f8cf5582"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:46:12.376261 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.376236 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-metrics-certs" pod "dd043f3c-4e08-4d7c-9c53-2260f8cf5582" (UID: "dd043f3c-4e08-4d7c-9c53-2260f8cf5582"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:46:12.376454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.376339 5838 generic.go:356] "Generic (PLEG): container finished" podID="dd043f3c-4e08-4d7c-9c53-2260f8cf5582" containerID="f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35" exitCode=0 Oct 30 16:46:12.376454 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.376435 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" event={"ID":"dd043f3c-4e08-4d7c-9c53-2260f8cf5582","Type":"ContainerDied","Data":"f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35"} Oct 30 16:46:12.376576 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.376547 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" event={"ID":"dd043f3c-4e08-4d7c-9c53-2260f8cf5582","Type":"ContainerDied","Data":"5eaa19f93e6e82c5d7b65c1ed1373dc44aa42ebe9cd94d9bf0e0b1d0b0af9d7c"} Oct 30 16:46:12.376632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.376586 5838 scope.go:122] "RemoveContainer" containerID="f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35" Oct 30 16:46:12.376798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.376779 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5" Oct 30 16:46:12.386636 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.386559 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-kube-api-access-qp7rt" pod "dd043f3c-4e08-4d7c-9c53-2260f8cf5582" (UID: "dd043f3c-4e08-4d7c-9c53-2260f8cf5582"). InnerVolumeSpecName "kube-api-access-qp7rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:12.399577 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.399550 5838 scope.go:122] "RemoveContainer" containerID="f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35" Oct 30 16:46:12.400002 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:46:12.399950 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35\": container with ID starting with f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35 not found: ID does not exist" containerID="f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35" Oct 30 16:46:12.400154 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.400014 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35"} err="failed to get container status \"f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35\": rpc error: code = NotFound desc = could not find container \"f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35\": container with ID starting with f7737b2e5f5a9f9da37f1c347055405b9870630ced189784c9a42cf6d0848f35 not found: ID does not exist" Oct 30 16:46:12.475083 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.475048 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-maas-api-tls\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:12.475249 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.475097 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7rt\" (UniqueName: \"kubernetes.io/projected/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-kube-api-access-qp7rt\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:12.475249 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.475107 5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd043f3c-4e08-4d7c-9c53-2260f8cf5582-metrics-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:12.716007 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.715883 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5"] Oct 30 16:46:12.721938 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:12.721911 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3bc669-7854cc49bd-n66w5"] Oct 30 16:46:13.766060 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:13.766016 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" podUID="4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f" containerName="payload-pre-processing" containerID="cri-o://08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6" gracePeriod=25 Oct 30 16:46:14.006717 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:14.006670 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:46:14.394733 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:14.394700 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd043f3c-4e08-4d7c-9c53-2260f8cf5582" path="/var/lib/kubelet/pods/dd043f3c-4e08-4d7c-9c53-2260f8cf5582/volumes" Oct 30 16:46:16.034417 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:16.034381 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q"] Oct 30 16:46:16.045328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:16.045296 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kfl8q"] Oct 30 16:46:16.399943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:16.399852 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d607990-f7c6-4317-9a6c-a3b1d22fc0a8" path="/var/lib/kubelet/pods/6d607990-f7c6-4317-9a6c-a3b1d22fc0a8/volumes" Oct 30 16:46:18.092783 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.092729 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" podUID="7891ff33-b413-4506-81cb-bc99d3a20c8e" containerName="main" containerID="cri-o://80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1" gracePeriod=45 Oct 30 16:46:18.130193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.130161 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" podUID="7891ff33-b413-4506-81cb-bc99d3a20c8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.160:8000/ready\": dial tcp 10.134.0.160:8000: connect: connection refused" Oct 30 16:46:18.346191 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.346126 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:46:18.418131 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.418093 5838 generic.go:356] "Generic (PLEG): container finished" podID="7891ff33-b413-4506-81cb-bc99d3a20c8e" containerID="80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1" exitCode=0 Oct 30 16:46:18.418344 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.418162 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" Oct 30 16:46:18.418344 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.418213 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" event={"ID":"7891ff33-b413-4506-81cb-bc99d3a20c8e","Type":"ContainerDied","Data":"80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1"} Oct 30 16:46:18.418344 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.418291 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d" event={"ID":"7891ff33-b413-4506-81cb-bc99d3a20c8e","Type":"ContainerDied","Data":"2ca495bbad05a0cb25e7730bcf057e453df30e3a4402da421f38e0a4842f10b6"} Oct 30 16:46:18.418344 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.418309 5838 scope.go:122] "RemoveContainer" containerID="80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1" Oct 30 16:46:18.432158 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.431700 5838 scope.go:122] "RemoveContainer" containerID="80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1" Oct 30 16:46:18.434427 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:46:18.434394 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1\": container with ID starting with 80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1 not found: ID does not exist" containerID="80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1" Oct 30 16:46:18.434552 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.434435 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1"} err="failed to get container status \"80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1\": rpc error: code = NotFound desc = could not find container \"80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1\": container with ID starting with 80c39ca212dae9db9964e30b0d2c883108f9fce887538a461dfdac3bfa6e7fa1 not found: ID does not exist" Oct 30 16:46:18.443282 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.443257 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-tmp-dir\") pod \"7891ff33-b413-4506-81cb-bc99d3a20c8e\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " Oct 30 16:46:18.443427 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.443333 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-home\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-home\") pod \"7891ff33-b413-4506-81cb-bc99d3a20c8e\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " Oct 30 16:46:18.443427 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.443395 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-dshm\") pod \"7891ff33-b413-4506-81cb-bc99d3a20c8e\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " Oct 30 16:46:18.443549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.443446 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-model-cache\") pod \"7891ff33-b413-4506-81cb-bc99d3a20c8e\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " Oct 30 16:46:18.443549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.443541 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7891ff33-b413-4506-81cb-bc99d3a20c8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/7891ff33-b413-4506-81cb-bc99d3a20c8e-tls-certs\") pod \"7891ff33-b413-4506-81cb-bc99d3a20c8e\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " Oct 30 16:46:18.443666 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.443565 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-tmp-dir" pod "7891ff33-b413-4506-81cb-bc99d3a20c8e" (UID: "7891ff33-b413-4506-81cb-bc99d3a20c8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:18.443666 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.443572 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7891ff33-b413-4506-81cb-bc99d3a20c8e-kube-api-access-wm9ml\" (UniqueName: \"kubernetes.io/projected/7891ff33-b413-4506-81cb-bc99d3a20c8e-kube-api-access-wm9ml\") pod \"7891ff33-b413-4506-81cb-bc99d3a20c8e\" (UID: \"7891ff33-b413-4506-81cb-bc99d3a20c8e\") " Oct 30 16:46:18.443666 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.443615 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-home" pod "7891ff33-b413-4506-81cb-bc99d3a20c8e" (UID: "7891ff33-b413-4506-81cb-bc99d3a20c8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:18.443843 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.443704 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-model-cache" pod "7891ff33-b413-4506-81cb-bc99d3a20c8e" (UID: "7891ff33-b413-4506-81cb-bc99d3a20c8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:18.444104 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.444084 5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-tmp-dir\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:18.444104 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.444105 5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-home\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:18.444265 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.444114 5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-model-cache\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:18.445601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.445578 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-dshm" pod "7891ff33-b413-4506-81cb-bc99d3a20c8e" (UID: "7891ff33-b413-4506-81cb-bc99d3a20c8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:18.445744 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.445722 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7891ff33-b413-4506-81cb-bc99d3a20c8e-tls-certs" pod "7891ff33-b413-4506-81cb-bc99d3a20c8e" (UID: "7891ff33-b413-4506-81cb-bc99d3a20c8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:46:18.446219 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.446194 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7891ff33-b413-4506-81cb-bc99d3a20c8e-kube-api-access-wm9ml" pod "7891ff33-b413-4506-81cb-bc99d3a20c8e" (UID: "7891ff33-b413-4506-81cb-bc99d3a20c8e"). InnerVolumeSpecName "kube-api-access-wm9ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:18.544605 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.544567 5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7891ff33-b413-4506-81cb-bc99d3a20c8e-dshm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:18.544605 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.544598 5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7891ff33-b413-4506-81cb-bc99d3a20c8e-tls-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:18.544605 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.544610 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm9ml\" (UniqueName: \"kubernetes.io/projected/7891ff33-b413-4506-81cb-bc99d3a20c8e-kube-api-access-wm9ml\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:18.762621 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.762589 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d"] Oct 30 16:46:18.774256 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:18.774227 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-3bc669/simulator-single-main-3bc669-kserve-5d4fc8ccb7-zf72d"] Oct 30 16:46:19.456567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:19.456528 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826"] Oct 30 16:46:19.457086 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:19.456792 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" containerID="cri-o://2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59" gracePeriod=30 Oct 30 16:46:19.933860 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:19.933826 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" podUID="4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:46:20.394509 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.394477 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7891ff33-b413-4506-81cb-bc99d3a20c8e" path="/var/lib/kubelet/pods/7891ff33-b413-4506-81cb-bc99d3a20c8e/volumes" Oct 30 16:46:20.721619 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.721520 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr"] Oct 30 16:46:20.732655 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.732624 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jxkrr"] Oct 30 16:46:20.779482 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.779450 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx"] Oct 30 16:46:20.779926 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.779911 5838 cpu_manager.go:414] "containerMap: removing container" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" containerName="cleanup" Oct 30 16:46:20.780003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.779927 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" containerName="cleanup" Oct 30 16:46:20.780003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.779949 5838 cpu_manager.go:414] "containerMap: removing container" podUID="dd043f3c-4e08-4d7c-9c53-2260f8cf5582" containerName="maas-api" Oct 30 16:46:20.780003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.779954 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd043f3c-4e08-4d7c-9c53-2260f8cf5582" containerName="maas-api" Oct 30 16:46:20.780003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.779963 5838 cpu_manager.go:414] "containerMap: removing container" podUID="7891ff33-b413-4506-81cb-bc99d3a20c8e" containerName="main" Oct 30 16:46:20.780003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.779980 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7891ff33-b413-4506-81cb-bc99d3a20c8e" containerName="main" Oct 30 16:46:20.780163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.780047 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="de62b9bf-270e-45c5-9444-0bb9fe1875be" containerName="cleanup" Oct 30 16:46:20.780163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.780061 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd043f3c-4e08-4d7c-9c53-2260f8cf5582" containerName="maas-api" Oct 30 16:46:20.780163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.780068 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="7891ff33-b413-4506-81cb-bc99d3a20c8e" containerName="main" Oct 30 16:46:20.784634 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.784617 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:20.805000 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.804954 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:46:20.806193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.806166 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:46:20.806335 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.806214 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:46:20.806601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.806577 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx"] Oct 30 16:46:20.810589 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.810562 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:46:20.869832 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.869799 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-thntx\" (UID: \"97d8cca2-4bc2-4f10-8329-5a809efed30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:20.870013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.869863 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcqpm\" (UniqueName: \"kubernetes.io/projected/97d8cca2-4bc2-4f10-8329-5a809efed30e-kube-api-access-jcqpm\") pod \"maas-api-revoke-sub-3d4156f1-thntx\" (UID: \"97d8cca2-4bc2-4f10-8329-5a809efed30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:20.870013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.869956 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-thntx\" (UID: \"97d8cca2-4bc2-4f10-8329-5a809efed30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:20.971088 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.971054 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-thntx\" (UID: \"97d8cca2-4bc2-4f10-8329-5a809efed30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:20.971255 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.971152 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-thntx\" (UID: \"97d8cca2-4bc2-4f10-8329-5a809efed30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:20.971255 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.971196 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcqpm\" (UniqueName: \"kubernetes.io/projected/97d8cca2-4bc2-4f10-8329-5a809efed30e-kube-api-access-jcqpm\") pod \"maas-api-revoke-sub-3d4156f1-thntx\" (UID: \"97d8cca2-4bc2-4f10-8329-5a809efed30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:20.971930 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.971859 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-thntx\" (UID: \"97d8cca2-4bc2-4f10-8329-5a809efed30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:20.973542 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.973517 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-thntx\" (UID: \"97d8cca2-4bc2-4f10-8329-5a809efed30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:20.993912 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:20.993881 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcqpm\" (UniqueName: \"kubernetes.io/projected/97d8cca2-4bc2-4f10-8329-5a809efed30e-kube-api-access-jcqpm\") pod \"maas-api-revoke-sub-3d4156f1-thntx\" (UID: \"97d8cca2-4bc2-4f10-8329-5a809efed30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:21.098559 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:21.098522 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:21.398712 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:21.398678 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx"] Oct 30 16:46:21.402136 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:46:21.402107 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d8cca2_4bc2_4f10_8329_5a809efed30e.slice/crio-8d2c7e5c53ab0516db7a99859dba30cbb60f4e880cf07879be073c7b44c60ae1 WatchSource:0}: Error finding container 8d2c7e5c53ab0516db7a99859dba30cbb60f4e880cf07879be073c7b44c60ae1: Status 404 returned error can't find the container with id 8d2c7e5c53ab0516db7a99859dba30cbb60f4e880cf07879be073c7b44c60ae1 Oct 30 16:46:21.441677 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:21.441645 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" event={"ID":"97d8cca2-4bc2-4f10-8329-5a809efed30e","Type":"ContainerStarted","Data":"8d2c7e5c53ab0516db7a99859dba30cbb60f4e880cf07879be073c7b44c60ae1"} Oct 30 16:46:22.394335 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:22.394304 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372ff1e4-0bac-457a-ab66-f174fd034767" path="/var/lib/kubelet/pods/372ff1e4-0bac-457a-ab66-f174fd034767/volumes" Oct 30 16:46:22.448784 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:22.448744 5838 generic.go:356] "Generic (PLEG): container finished" podID="97d8cca2-4bc2-4f10-8329-5a809efed30e" containerID="ae502831374f96a840892c51cca3192731278818ea7204da70f7e226bf87cc83" exitCode=0 Oct 30 16:46:22.449007 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:22.448827 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" event={"ID":"97d8cca2-4bc2-4f10-8329-5a809efed30e","Type":"ContainerDied","Data":"ae502831374f96a840892c51cca3192731278818ea7204da70f7e226bf87cc83"} Oct 30 16:46:23.595642 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:23.595617 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:23.699836 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:23.699805 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-openshift-ca\") pod \"97d8cca2-4bc2-4f10-8329-5a809efed30e\" (UID: \"97d8cca2-4bc2-4f10-8329-5a809efed30e\") " Oct 30 16:46:23.700042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:23.699860 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d8cca2-4bc2-4f10-8329-5a809efed30e-kube-api-access-jcqpm\" (UniqueName: \"kubernetes.io/projected/97d8cca2-4bc2-4f10-8329-5a809efed30e-kube-api-access-jcqpm\") pod \"97d8cca2-4bc2-4f10-8329-5a809efed30e\" (UID: \"97d8cca2-4bc2-4f10-8329-5a809efed30e\") " Oct 30 16:46:23.700042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:23.699935 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-xks-ca\") pod \"97d8cca2-4bc2-4f10-8329-5a809efed30e\" (UID: \"97d8cca2-4bc2-4f10-8329-5a809efed30e\") " Oct 30 16:46:23.700301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:23.700273 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-openshift-ca" pod "97d8cca2-4bc2-4f10-8329-5a809efed30e" (UID: "97d8cca2-4bc2-4f10-8329-5a809efed30e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:23.702073 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:23.702049 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-xks-ca" pod "97d8cca2-4bc2-4f10-8329-5a809efed30e" (UID: "97d8cca2-4bc2-4f10-8329-5a809efed30e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:46:23.702191 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:23.702054 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d8cca2-4bc2-4f10-8329-5a809efed30e-kube-api-access-jcqpm" pod "97d8cca2-4bc2-4f10-8329-5a809efed30e" (UID: "97d8cca2-4bc2-4f10-8329-5a809efed30e"). InnerVolumeSpecName "kube-api-access-jcqpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:23.801645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:23.801611 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:23.801645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:23.801638 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcqpm\" (UniqueName: \"kubernetes.io/projected/97d8cca2-4bc2-4f10-8329-5a809efed30e-kube-api-access-jcqpm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:23.801645 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:23.801649 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97d8cca2-4bc2-4f10-8329-5a809efed30e-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:24.461943 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.461909 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" Oct 30 16:46:24.462147 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.461941 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx" event={"ID":"97d8cca2-4bc2-4f10-8329-5a809efed30e","Type":"ContainerDied","Data":"8d2c7e5c53ab0516db7a99859dba30cbb60f4e880cf07879be073c7b44c60ae1"} Oct 30 16:46:24.462147 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.462003 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d2c7e5c53ab0516db7a99859dba30cbb60f4e880cf07879be073c7b44c60ae1" Oct 30 16:46:24.715516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.715487 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:46:24.810193 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.810163 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-certs\") pod \"91179b4c-4588-46ac-af09-924ed33cac64\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " Oct 30 16:46:24.810372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.810203 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-istio-token\" (UniqueName: \"kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-istio-token\") pod \"91179b4c-4588-46ac-af09-924ed33cac64\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " Oct 30 16:46:24.810372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.810243 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91179b4c-4588-46ac-af09-924ed33cac64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91179b4c-4588-46ac-af09-924ed33cac64-istio-podinfo\") pod \"91179b4c-4588-46ac-af09-924ed33cac64\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " Oct 30 16:46:24.810372 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.810276 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-data\") pod \"91179b4c-4588-46ac-af09-924ed33cac64\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " Oct 30 16:46:24.810536 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.810456 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-kube-api-access-99cm2\" (UniqueName: \"kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-kube-api-access-99cm2\") pod \"91179b4c-4588-46ac-af09-924ed33cac64\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " Oct 30 16:46:24.810536 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.810490 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-certs" pod "91179b4c-4588-46ac-af09-924ed33cac64" (UID: "91179b4c-4588-46ac-af09-924ed33cac64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:24.810642 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.810588 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-data" pod "91179b4c-4588-46ac-af09-924ed33cac64" (UID: "91179b4c-4588-46ac-af09-924ed33cac64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:24.810642 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.810591 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-envoy\") pod \"91179b4c-4588-46ac-af09-924ed33cac64\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " Oct 30 16:46:24.810642 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.810628 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-credential-socket\") pod \"91179b4c-4588-46ac-af09-924ed33cac64\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " Oct 30 16:46:24.810799 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.810668 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91179b4c-4588-46ac-af09-924ed33cac64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91179b4c-4588-46ac-af09-924ed33cac64-istiod-ca-cert\") pod \"91179b4c-4588-46ac-af09-924ed33cac64\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " Oct 30 16:46:24.810799 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.810720 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-socket\") pod \"91179b4c-4588-46ac-af09-924ed33cac64\" (UID: \"91179b4c-4588-46ac-af09-924ed33cac64\") " Oct 30 16:46:24.810899 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.810880 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-credential-socket" pod "91179b4c-4588-46ac-af09-924ed33cac64" (UID: "91179b4c-4588-46ac-af09-924ed33cac64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:24.811141 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.811095 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:24.811141 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.811121 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:24.811141 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.811139 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:24.811322 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.811119 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-socket" pod "91179b4c-4588-46ac-af09-924ed33cac64" (UID: "91179b4c-4588-46ac-af09-924ed33cac64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:24.811322 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.811153 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91179b4c-4588-46ac-af09-924ed33cac64-istiod-ca-cert" pod "91179b4c-4588-46ac-af09-924ed33cac64" (UID: "91179b4c-4588-46ac-af09-924ed33cac64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:24.812618 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.812589 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-envoy" pod "91179b4c-4588-46ac-af09-924ed33cac64" (UID: "91179b4c-4588-46ac-af09-924ed33cac64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:46:24.812618 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.812610 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-istio-token" pod "91179b4c-4588-46ac-af09-924ed33cac64" (UID: "91179b4c-4588-46ac-af09-924ed33cac64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:24.812863 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.812849 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91179b4c-4588-46ac-af09-924ed33cac64-istio-podinfo" pod "91179b4c-4588-46ac-af09-924ed33cac64" (UID: "91179b4c-4588-46ac-af09-924ed33cac64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:46:24.812913 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.812864 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-kube-api-access-99cm2" pod "91179b4c-4588-46ac-af09-924ed33cac64" (UID: "91179b4c-4588-46ac-af09-924ed33cac64"). InnerVolumeSpecName "kube-api-access-99cm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:24.912070 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.912033 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:24.912070 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.912061 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91179b4c-4588-46ac-af09-924ed33cac64-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:24.912070 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.912070 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91179b4c-4588-46ac-af09-924ed33cac64-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:24.912070 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.912079 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:24.912336 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.912087 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91179b4c-4588-46ac-af09-924ed33cac64-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:24.912336 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:24.912096 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99cm2\" (UniqueName: \"kubernetes.io/projected/91179b4c-4588-46ac-af09-924ed33cac64-kube-api-access-99cm2\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:25.469527 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:25.469485 5838 generic.go:356] "Generic (PLEG): container finished" podID="91179b4c-4588-46ac-af09-924ed33cac64" containerID="2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59" exitCode=0 Oct 30 16:46:25.469696 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:25.469557 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" Oct 30 16:46:25.469696 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:25.469567 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" event={"ID":"91179b4c-4588-46ac-af09-924ed33cac64","Type":"ContainerDied","Data":"2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59"} Oct 30 16:46:25.469696 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:25.469622 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826" event={"ID":"91179b4c-4588-46ac-af09-924ed33cac64","Type":"ContainerDied","Data":"07646e1e5a82f38025d2936108420dd3a2e56b85e5a73b90e084f6ebdf25cc11"} Oct 30 16:46:25.469696 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:25.469648 5838 scope.go:122] "RemoveContainer" containerID="2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59" Oct 30 16:46:25.480051 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:25.480032 5838 scope.go:122] "RemoveContainer" containerID="8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34" Oct 30 16:46:25.489106 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:25.489085 5838 scope.go:122] "RemoveContainer" containerID="2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59" Oct 30 16:46:25.489384 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:46:25.489362 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59\": container with ID starting with 2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59 not found: ID does not exist" containerID="2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59" Oct 30 16:46:25.489460 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:25.489400 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59"} err="failed to get container status \"2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59\": rpc error: code = NotFound desc = could not find container \"2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59\": container with ID starting with 2405eb56a0a3d109e1c34dc3b8680af58a558a69ad66e26d4881da5ebaf28e59 not found: ID does not exist" Oct 30 16:46:25.489460 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:25.489427 5838 scope.go:122] "RemoveContainer" containerID="8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34" Oct 30 16:46:25.489681 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:46:25.489663 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34\": container with ID starting with 8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34 not found: ID does not exist" containerID="8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34" Oct 30 16:46:25.489719 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:25.489689 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34"} err="failed to get container status \"8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34\": rpc error: code = NotFound desc = could not find container \"8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34\": container with ID starting with 8c3d69d0660c1d053625c76a4a3b5e53a1d573833bb6472ea52dd37881546b34 not found: ID does not exist" Oct 30 16:46:25.531824 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:25.531791 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826"] Oct 30 16:46:25.535545 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:25.535520 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3bc669-openshift-default-5bdc44896fd4826"] Oct 30 16:46:26.394880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:26.394849 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91179b4c-4588-46ac-af09-924ed33cac64" path="/var/lib/kubelet/pods/91179b4c-4588-46ac-af09-924ed33cac64/volumes" Oct 30 16:46:29.933941 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:29.933901 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" podUID="4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:46:31.796283 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:46:31.796244 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:46:32.757704 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:32.757670 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:46:34.287572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.287193 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" Oct 30 16:46:34.306515 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.306489 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-kube-api-access-ml82d\" (UniqueName: \"kubernetes.io/projected/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-kube-api-access-ml82d\") pod \"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f\" (UID: \"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f\") " Oct 30 16:46:34.306657 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.306529 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-plugins-config-volume\") pod \"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f\" (UID: \"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f\") " Oct 30 16:46:34.307079 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.307047 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-plugins-config-volume" pod "4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f" (UID: "4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:34.308812 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.308787 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-kube-api-access-ml82d" pod "4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f" (UID: "4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f"). InnerVolumeSpecName "kube-api-access-ml82d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:34.407316 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.407285 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml82d\" (UniqueName: \"kubernetes.io/projected/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-kube-api-access-ml82d\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:34.407316 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.407314 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:46:34.586911 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.585639 5838 generic.go:356] "Generic (PLEG): container finished" podID="4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f" containerID="08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6" exitCode=0 Oct 30 16:46:34.586911 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.585727 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" event={"ID":"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f","Type":"ContainerDied","Data":"08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6"} Oct 30 16:46:34.586911 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.585781 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" event={"ID":"4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f","Type":"ContainerDied","Data":"20c849f7f695d529cdc1be923b5647c31cd883b03adab53a5cf5a638dca00ff8"} Oct 30 16:46:34.586911 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.585815 5838 scope.go:122] "RemoveContainer" containerID="08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6" Oct 30 16:46:34.586911 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.586002 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv" Oct 30 16:46:34.602161 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.601965 5838 scope.go:122] "RemoveContainer" containerID="08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6" Oct 30 16:46:34.602778 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:46:34.602754 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6\": container with ID starting with 08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6 not found: ID does not exist" containerID="08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6" Oct 30 16:46:34.602898 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.602784 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6"} err="failed to get container status \"08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6\": rpc error: code = NotFound desc = could not find container \"08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6\": container with ID starting with 08687db1c3697024c09c79689b49383b714384320f60383bd85542c38cc1a9c6 not found: ID does not exist" Oct 30 16:46:34.663994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.663450 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv"] Oct 30 16:46:34.686477 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:34.686364 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3bc669-99f67df2xv"] Oct 30 16:46:36.030841 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:36.030804 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w"] Oct 30 16:46:36.042471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:36.042435 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-j7j7w"] Oct 30 16:46:36.396585 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:36.396505 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f" path="/var/lib/kubelet/pods/4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f/volumes" Oct 30 16:46:36.397119 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:36.397102 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7dd338-d668-4bd3-8fa3-41a8ab062a8c" path="/var/lib/kubelet/pods/da7dd338-d668-4bd3-8fa3-41a8ab062a8c/volumes" Oct 30 16:46:38.140334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:38.140293 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:46:40.784229 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:40.784190 5838 scope.go:122] "RemoveContainer" containerID="98ea9115b6e05b417f98cc68a655d49d9c7588bb7f401f312cbed6fc7f844992" Oct 30 16:46:47.030235 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:47.030195 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp"] Oct 30 16:46:47.041880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:47.041850 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4bcmp"] Oct 30 16:46:48.393750 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:46:48.393714 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2329056-283e-4b01-8b5c-61c3ef288f99" path="/var/lib/kubelet/pods/e2329056-283e-4b01-8b5c-61c3ef288f99/volumes" Oct 30 16:47:01.795750 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:47:01.795702 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:47:14.026559 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:47:14.026522 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm"] Oct 30 16:47:14.037757 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:47:14.037716 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b7mtm"] Oct 30 16:47:14.394572 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:47:14.394492 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5246c38-187e-4be4-ad87-990d95716d1f" path="/var/lib/kubelet/pods/d5246c38-187e-4be4-ad87-990d95716d1f/volumes" Oct 30 16:47:24.029399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:47:24.029337 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75"] Oct 30 16:47:24.040733 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:47:24.040697 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-9qn75"] Oct 30 16:47:24.394796 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:47:24.394709 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a204b7f6-1772-45f7-a2de-946dd200d134" path="/var/lib/kubelet/pods/a204b7f6-1772-45f7-a2de-946dd200d134/volumes" Oct 30 16:47:31.796703 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:47:31.796661 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:47:40.827142 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:47:40.827104 5838 scope.go:122] "RemoveContainer" containerID="2f81f9ff5b444ac7c4981ee5e18637aeec192c4b40567c8f404068a6d2f49ad9" Oct 30 16:47:40.836539 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:47:40.836516 5838 scope.go:122] "RemoveContainer" containerID="6735d54ac5a4dda0807f6e9fc7e1b26520c2cfb1679c8411ec382c9007bcae87" Oct 30 16:47:47.055248 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:47:47.055205 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r"] Oct 30 16:47:47.066340 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:47:47.066310 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6vn7r"] Oct 30 16:47:48.393930 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:47:48.393895 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff72534d-dc5b-4a4c-8489-6650f0ba6a8f" path="/var/lib/kubelet/pods/ff72534d-dc5b-4a4c-8489-6650f0ba6a8f/volumes" Oct 30 16:48:01.796445 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:48:01.796366 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:48:05.922526 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:05.922487 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:48:09.028033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.027991 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47"] Oct 30 16:48:09.039119 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.039080 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7sn47"] Oct 30 16:48:09.929951 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.929917 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4"] Oct 30 16:48:09.930586 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.930559 5838 cpu_manager.go:414] "containerMap: removing container" podUID="97d8cca2-4bc2-4f10-8329-5a809efed30e" containerName="revoke-keys" Oct 30 16:48:09.930586 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.930583 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d8cca2-4bc2-4f10-8329-5a809efed30e" containerName="revoke-keys" Oct 30 16:48:09.930724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.930595 5838 cpu_manager.go:414] "containerMap: removing container" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" Oct 30 16:48:09.930724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.930602 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" Oct 30 16:48:09.930724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.930610 5838 cpu_manager.go:414] "containerMap: removing container" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" Oct 30 16:48:09.930724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.930616 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" Oct 30 16:48:09.930724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.930625 5838 cpu_manager.go:414] "containerMap: removing container" podUID="4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f" containerName="payload-pre-processing" Oct 30 16:48:09.930724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.930630 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f" containerName="payload-pre-processing" Oct 30 16:48:09.930724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.930711 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d8cca2-4bc2-4f10-8329-5a809efed30e" containerName="revoke-keys" Oct 30 16:48:09.930724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.930719 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" Oct 30 16:48:09.930724 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.930727 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="91179b4c-4588-46ac-af09-924ed33cac64" containerName="istio-proxy" Oct 30 16:48:09.931038 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.930734 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa30de4-1e7a-4cf1-8b88-c6f2d66df65f" containerName="payload-pre-processing" Oct 30 16:48:09.933762 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.933741 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:09.953170 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.953136 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4"] Oct 30 16:48:09.962668 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.962635 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:48:09.962860 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.962662 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:48:09.962860 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.962733 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:48:09.962860 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:09.962752 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:48:10.016749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:10.016710 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8q2b4\" (UID: \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:10.016749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:10.016753 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ztb9\" (UniqueName: \"kubernetes.io/projected/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-kube-api-access-5ztb9\") pod \"maas-api-revoke-sub-d576e68b-8q2b4\" (UID: \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:10.016961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:10.016875 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8q2b4\" (UID: \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:10.118272 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:10.118229 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8q2b4\" (UID: \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:10.118272 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:10.118273 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ztb9\" (UniqueName: \"kubernetes.io/projected/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-kube-api-access-5ztb9\") pod \"maas-api-revoke-sub-d576e68b-8q2b4\" (UID: \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:10.118713 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:10.118326 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8q2b4\" (UID: \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:10.119018 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:10.118994 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8q2b4\" (UID: \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:10.120706 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:10.120679 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8q2b4\" (UID: \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:10.141357 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:10.141320 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ztb9\" (UniqueName: \"kubernetes.io/projected/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-kube-api-access-5ztb9\") pod \"maas-api-revoke-sub-d576e68b-8q2b4\" (UID: \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:10.247514 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:10.247481 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:10.394830 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:10.394795 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72042b0-a36d-4079-9d08-712865ac0660" path="/var/lib/kubelet/pods/b72042b0-a36d-4079-9d08-712865ac0660/volumes" Oct 30 16:48:10.550528 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:10.550430 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4"] Oct 30 16:48:10.554300 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:48:10.554271 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78dd4f4_4b65_4a24_9cc1_497f54756e2d.slice/crio-f817e0f19bd7861ba1fa79bcdf147752d9057ebd601aa468395c25ad993e782d WatchSource:0}: Error finding container f817e0f19bd7861ba1fa79bcdf147752d9057ebd601aa468395c25ad993e782d: Status 404 returned error can't find the container with id f817e0f19bd7861ba1fa79bcdf147752d9057ebd601aa468395c25ad993e782d Oct 30 16:48:11.181097 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:11.181005 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" event={"ID":"e78dd4f4-4b65-4a24-9cc1-497f54756e2d","Type":"ContainerStarted","Data":"9d3d1e2d420e6d1fa4d7ea1706a3e236eaedb6c7d60944259f16f355616d771c"} Oct 30 16:48:11.181097 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:11.181084 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" event={"ID":"e78dd4f4-4b65-4a24-9cc1-497f54756e2d","Type":"ContainerStarted","Data":"f817e0f19bd7861ba1fa79bcdf147752d9057ebd601aa468395c25ad993e782d"} Oct 30 16:48:11.219334 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:11.219271 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" podStartSLOduration=2.219257046 podStartE2EDuration="2.219257046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:11.218783271 +0000 UTC m=+1652.967372681" watchObservedRunningTime="2026-10-30 16:48:11.219257046 +0000 UTC m=+1652.967846435" Oct 30 16:48:12.188669 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:12.188634 5838 generic.go:356] "Generic (PLEG): container finished" podID="e78dd4f4-4b65-4a24-9cc1-497f54756e2d" containerID="9d3d1e2d420e6d1fa4d7ea1706a3e236eaedb6c7d60944259f16f355616d771c" exitCode=0 Oct 30 16:48:12.189093 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:12.188729 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" event={"ID":"e78dd4f4-4b65-4a24-9cc1-497f54756e2d","Type":"ContainerDied","Data":"9d3d1e2d420e6d1fa4d7ea1706a3e236eaedb6c7d60944259f16f355616d771c"} Oct 30 16:48:13.334705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:13.334676 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:13.447303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:13.447215 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-kube-api-access-5ztb9\" (UniqueName: \"kubernetes.io/projected/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-kube-api-access-5ztb9\") pod \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\" (UID: \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\") " Oct 30 16:48:13.447303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:13.447284 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-openshift-ca\") pod \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\" (UID: \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\") " Oct 30 16:48:13.447563 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:13.447343 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-xks-ca\") pod \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\" (UID: \"e78dd4f4-4b65-4a24-9cc1-497f54756e2d\") " Oct 30 16:48:13.447698 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:13.447674 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-openshift-ca" pod "e78dd4f4-4b65-4a24-9cc1-497f54756e2d" (UID: "e78dd4f4-4b65-4a24-9cc1-497f54756e2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:13.449392 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:13.449366 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-xks-ca" pod "e78dd4f4-4b65-4a24-9cc1-497f54756e2d" (UID: "e78dd4f4-4b65-4a24-9cc1-497f54756e2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:48:13.449456 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:13.449379 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-kube-api-access-5ztb9" pod "e78dd4f4-4b65-4a24-9cc1-497f54756e2d" (UID: "e78dd4f4-4b65-4a24-9cc1-497f54756e2d"). InnerVolumeSpecName "kube-api-access-5ztb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:13.548426 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:13.548383 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:48:13.548426 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:13.548420 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ztb9\" (UniqueName: \"kubernetes.io/projected/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-kube-api-access-5ztb9\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:48:13.548426 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:13.548431 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e78dd4f4-4b65-4a24-9cc1-497f54756e2d-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.201806 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:14.201774 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" Oct 30 16:48:14.202021 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:14.201814 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4" event={"ID":"e78dd4f4-4b65-4a24-9cc1-497f54756e2d","Type":"ContainerDied","Data":"f817e0f19bd7861ba1fa79bcdf147752d9057ebd601aa468395c25ad993e782d"} Oct 30 16:48:14.202021 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:14.201871 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f817e0f19bd7861ba1fa79bcdf147752d9057ebd601aa468395c25ad993e782d" Oct 30 16:48:21.310593 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:21.310556 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:48:26.878037 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:26.878000 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn"] Oct 30 16:48:26.878635 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:26.878485 5838 cpu_manager.go:414] "containerMap: removing container" podUID="e78dd4f4-4b65-4a24-9cc1-497f54756e2d" containerName="revoke-keys" Oct 30 16:48:26.878635 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:26.878502 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78dd4f4-4b65-4a24-9cc1-497f54756e2d" containerName="revoke-keys" Oct 30 16:48:26.878635 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:26.878625 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78dd4f4-4b65-4a24-9cc1-497f54756e2d" containerName="revoke-keys" Oct 30 16:48:26.881695 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:26.881678 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:26.898608 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:26.898548 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:48:26.900927 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:26.900904 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn"] Oct 30 16:48:26.909785 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:26.909758 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:48:26.909984 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:26.909774 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:48:26.909984 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:26.909841 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:48:27.000689 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:27.000655 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4w5p\" (UniqueName: \"kubernetes.io/projected/18bd0d59-647e-4683-9019-82d6a95ade7b-kube-api-access-r4w5p\") pod \"maas-api-revoke-sub-d2bc93bc-bjvdn\" (UID: \"18bd0d59-647e-4683-9019-82d6a95ade7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:27.000861 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:27.000710 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bjvdn\" (UID: \"18bd0d59-647e-4683-9019-82d6a95ade7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:27.000861 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:27.000767 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bjvdn\" (UID: \"18bd0d59-647e-4683-9019-82d6a95ade7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:27.101293 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:27.101259 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bjvdn\" (UID: \"18bd0d59-647e-4683-9019-82d6a95ade7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:27.101473 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:27.101302 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bjvdn\" (UID: \"18bd0d59-647e-4683-9019-82d6a95ade7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:27.101473 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:27.101416 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4w5p\" (UniqueName: \"kubernetes.io/projected/18bd0d59-647e-4683-9019-82d6a95ade7b-kube-api-access-r4w5p\") pod \"maas-api-revoke-sub-d2bc93bc-bjvdn\" (UID: \"18bd0d59-647e-4683-9019-82d6a95ade7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:27.102145 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:27.102113 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bjvdn\" (UID: \"18bd0d59-647e-4683-9019-82d6a95ade7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:27.103732 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:27.103696 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-bjvdn\" (UID: \"18bd0d59-647e-4683-9019-82d6a95ade7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:27.124483 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:27.124451 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4w5p\" (UniqueName: \"kubernetes.io/projected/18bd0d59-647e-4683-9019-82d6a95ade7b-kube-api-access-r4w5p\") pod \"maas-api-revoke-sub-d2bc93bc-bjvdn\" (UID: \"18bd0d59-647e-4683-9019-82d6a95ade7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:27.194315 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:27.194232 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:27.495657 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:27.495623 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn"] Oct 30 16:48:27.499832 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:48:27.499805 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bd0d59_647e_4683_9019_82d6a95ade7b.slice/crio-8389289d48f8921d2b6f5a13e5433eaec84cdd5a3e3e787b5d81998b9b08c840 WatchSource:0}: Error finding container 8389289d48f8921d2b6f5a13e5433eaec84cdd5a3e3e787b5d81998b9b08c840: Status 404 returned error can't find the container with id 8389289d48f8921d2b6f5a13e5433eaec84cdd5a3e3e787b5d81998b9b08c840 Oct 30 16:48:28.300473 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:28.300428 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" event={"ID":"18bd0d59-647e-4683-9019-82d6a95ade7b","Type":"ContainerStarted","Data":"d5a4267283b5cd55a122b379ee305e16ff59ab64576e8309d4524adcc3cc6de5"} Oct 30 16:48:28.300890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:28.300497 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" event={"ID":"18bd0d59-647e-4683-9019-82d6a95ade7b","Type":"ContainerStarted","Data":"8389289d48f8921d2b6f5a13e5433eaec84cdd5a3e3e787b5d81998b9b08c840"} Oct 30 16:48:28.350833 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:28.350788 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" podStartSLOduration=2.350775911 podStartE2EDuration="2.350775911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:28.338688499 +0000 UTC m=+1670.087277904" watchObservedRunningTime="2026-10-30 16:48:28.350775911 +0000 UTC m=+1670.099365299" Oct 30 16:48:29.307597 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:29.307560 5838 generic.go:356] "Generic (PLEG): container finished" podID="18bd0d59-647e-4683-9019-82d6a95ade7b" containerID="d5a4267283b5cd55a122b379ee305e16ff59ab64576e8309d4524adcc3cc6de5" exitCode=0 Oct 30 16:48:29.308046 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:29.307639 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" event={"ID":"18bd0d59-647e-4683-9019-82d6a95ade7b","Type":"ContainerDied","Data":"d5a4267283b5cd55a122b379ee305e16ff59ab64576e8309d4524adcc3cc6de5"} Oct 30 16:48:30.450529 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:30.450504 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:30.536123 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:30.536088 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18bd0d59-647e-4683-9019-82d6a95ade7b-kube-api-access-r4w5p\" (UniqueName: \"kubernetes.io/projected/18bd0d59-647e-4683-9019-82d6a95ade7b-kube-api-access-r4w5p\") pod \"18bd0d59-647e-4683-9019-82d6a95ade7b\" (UID: \"18bd0d59-647e-4683-9019-82d6a95ade7b\") " Oct 30 16:48:30.536325 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:30.536179 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-xks-ca\") pod \"18bd0d59-647e-4683-9019-82d6a95ade7b\" (UID: \"18bd0d59-647e-4683-9019-82d6a95ade7b\") " Oct 30 16:48:30.536325 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:30.536249 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-openshift-ca\") pod \"18bd0d59-647e-4683-9019-82d6a95ade7b\" (UID: \"18bd0d59-647e-4683-9019-82d6a95ade7b\") " Oct 30 16:48:30.536602 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:30.536569 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-openshift-ca" pod "18bd0d59-647e-4683-9019-82d6a95ade7b" (UID: "18bd0d59-647e-4683-9019-82d6a95ade7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:30.538271 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:30.538247 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-xks-ca" pod "18bd0d59-647e-4683-9019-82d6a95ade7b" (UID: "18bd0d59-647e-4683-9019-82d6a95ade7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:48:30.538382 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:30.538297 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bd0d59-647e-4683-9019-82d6a95ade7b-kube-api-access-r4w5p" pod "18bd0d59-647e-4683-9019-82d6a95ade7b" (UID: "18bd0d59-647e-4683-9019-82d6a95ade7b"). InnerVolumeSpecName "kube-api-access-r4w5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:30.637756 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:30.637670 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:48:30.637756 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:30.637697 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4w5p\" (UniqueName: \"kubernetes.io/projected/18bd0d59-647e-4683-9019-82d6a95ade7b-kube-api-access-r4w5p\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:48:30.637756 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:30.637709 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18bd0d59-647e-4683-9019-82d6a95ade7b-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:48:31.029296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:31.029265 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn"] Oct 30 16:48:31.040563 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:31.040532 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-h66vn"] Oct 30 16:48:31.320453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:31.320373 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" Oct 30 16:48:31.320453 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:31.320408 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn" event={"ID":"18bd0d59-647e-4683-9019-82d6a95ade7b","Type":"ContainerDied","Data":"8389289d48f8921d2b6f5a13e5433eaec84cdd5a3e3e787b5d81998b9b08c840"} Oct 30 16:48:31.320654 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:31.320458 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8389289d48f8921d2b6f5a13e5433eaec84cdd5a3e3e787b5d81998b9b08c840" Oct 30 16:48:31.797141 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:48:31.797103 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:48:32.394103 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:32.394072 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf3a0e2-29e7-4080-9f88-8da63aaa25ba" path="/var/lib/kubelet/pods/dcf3a0e2-29e7-4080-9f88-8da63aaa25ba/volumes" Oct 30 16:48:39.119526 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.119471 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b"] Oct 30 16:48:39.120281 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.120258 5838 cpu_manager.go:414] "containerMap: removing container" podUID="18bd0d59-647e-4683-9019-82d6a95ade7b" containerName="revoke-keys" Oct 30 16:48:39.120346 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.120285 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bd0d59-647e-4683-9019-82d6a95ade7b" containerName="revoke-keys" Oct 30 16:48:39.120421 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.120408 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bd0d59-647e-4683-9019-82d6a95ade7b" containerName="revoke-keys" Oct 30 16:48:39.124087 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.124059 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.142379 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.142350 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-3060bb-openshift-default-dockercfg-66r4b\"" Oct 30 16:48:39.153718 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.153685 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b"] Oct 30 16:48:39.229738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.229685 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s79nz\" (UniqueName: \"kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-kube-api-access-s79nz\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.230085 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.230056 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-podinfo\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.230257 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.230232 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-credential-socket\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.230397 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.230370 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-data\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.230547 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.230421 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-envoy\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.230547 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.230492 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istiod-ca-cert\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.230547 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.230541 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-socket\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.230732 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.230610 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-token\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.230732 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.230676 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-certs\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.331703 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.331660 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-socket\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.331920 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.331735 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-token\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.331920 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.331795 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-certs\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.331920 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.331829 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s79nz\" (UniqueName: \"kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-kube-api-access-s79nz\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.331920 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.331879 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-podinfo\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.331920 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.331907 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-credential-socket\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.332273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.331930 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-data\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.332273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.332002 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-envoy\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.332273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.332072 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istiod-ca-cert\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.332273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.332242 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-socket\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.332426 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.332289 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-certs\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.332426 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.332378 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-credential-socket\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.332643 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.332616 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-data\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.332771 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.332751 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istiod-ca-cert\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.334865 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.334844 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-podinfo\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.334987 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.334919 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-envoy\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.354632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.354596 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-token\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.354758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.354734 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s79nz\" (UniqueName: \"kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-kube-api-access-s79nz\") pod \"e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.439894 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.439809 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:39.763922 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.763767 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b"] Oct 30 16:48:39.766897 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:48:39.766868 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95fe69c_92eb_4331_b46a_3cbab2f9a864.slice/crio-f6ea3d4502399b19319a616ab87e0329718c9873168720d651bb742f14be87b3 WatchSource:0}: Error finding container f6ea3d4502399b19319a616ab87e0329718c9873168720d651bb742f14be87b3: Status 404 returned error can't find the container with id f6ea3d4502399b19319a616ab87e0329718c9873168720d651bb742f14be87b3 Oct 30 16:48:39.769145 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.769114 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:48:39.769232 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.769179 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:48:39.769232 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:39.769209 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:48:40.379741 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:40.379698 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" event={"ID":"d95fe69c-92eb-4331-b46a-3cbab2f9a864","Type":"ContainerStarted","Data":"5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8"} Oct 30 16:48:40.380218 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:40.379755 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" event={"ID":"d95fe69c-92eb-4331-b46a-3cbab2f9a864","Type":"ContainerStarted","Data":"f6ea3d4502399b19319a616ab87e0329718c9873168720d651bb742f14be87b3"} Oct 30 16:48:40.440814 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:40.440720 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:40.446072 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:40.446041 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:40.447474 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:40.447430 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" podStartSLOduration=1.447417599 podStartE2EDuration="1.447417599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:40.435587527 +0000 UTC m=+1682.184176933" watchObservedRunningTime="2026-10-30 16:48:40.447417599 +0000 UTC m=+1682.196006980" Oct 30 16:48:40.862275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:40.862239 5838 scope.go:122] "RemoveContainer" containerID="695ffb70044859ef31831115596eed0787148d58f673a6f7b8c18327d69f0efd" Oct 30 16:48:40.871015 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:40.870991 5838 scope.go:122] "RemoveContainer" containerID="2822164b8c3ccd1e7813b0e349cf0416339fa5ab499de82a9b13c2ff60b426fb" Oct 30 16:48:40.879474 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:40.879448 5838 scope.go:122] "RemoveContainer" containerID="b862db8c4825d8bfe518c27ccb5ecfd91ccc3c3eccdbeae630e45604725f98b1" Oct 30 16:48:41.385893 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:41.385859 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:41.386797 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:41.386780 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:48:44.114232 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.114182 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv"] Oct 30 16:48:44.119740 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.119710 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.144090 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.144053 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv"] Oct 30 16:48:44.149890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.149861 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3060bb-serving-cert\"" Oct 30 16:48:44.149890 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.149864 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dqwfr\"" Oct 30 16:48:44.150163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.149908 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 16:48:44.189323 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.189288 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:48:44.189497 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.189325 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:48:44.307516 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.307470 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg8r7\" (UniqueName: \"kubernetes.io/projected/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-kube-api-access-hg8r7\") pod \"maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.307883 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.307850 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.308115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.308096 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.308259 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.308221 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.409107 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.409001 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.409107 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.409059 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.409356 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.409213 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg8r7\" (UniqueName: \"kubernetes.io/projected/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-kube-api-access-hg8r7\") pod \"maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.409356 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.409315 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.410120 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.410092 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.412467 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.412438 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.412873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.412842 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.435467 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.435431 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg8r7\" (UniqueName: \"kubernetes.io/projected/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-kube-api-access-hg8r7\") pod \"maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.649078 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.649036 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc"] Oct 30 16:48:44.652985 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.652947 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" Oct 30 16:48:44.669635 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.669605 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-f6c91d3c-dockercfg-29q27\"" Oct 30 16:48:44.669762 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.669691 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3060bb\"" Oct 30 16:48:44.671924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.671900 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc"] Oct 30 16:48:44.711836 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.711797 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-httcj\" (UniqueName: \"kubernetes.io/projected/c47f0a31-b4dc-4648-88f4-3b54bf894a85-kube-api-access-httcj\") pod \"payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc\" (UID: \"c47f0a31-b4dc-4648-88f4-3b54bf894a85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" Oct 30 16:48:44.712030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.711844 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c47f0a31-b4dc-4648-88f4-3b54bf894a85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc\" (UID: \"c47f0a31-b4dc-4648-88f4-3b54bf894a85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" Oct 30 16:48:44.737527 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.737484 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:44.812739 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.812700 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httcj\" (UniqueName: \"kubernetes.io/projected/c47f0a31-b4dc-4648-88f4-3b54bf894a85-kube-api-access-httcj\") pod \"payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc\" (UID: \"c47f0a31-b4dc-4648-88f4-3b54bf894a85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" Oct 30 16:48:44.812918 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.812761 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c47f0a31-b4dc-4648-88f4-3b54bf894a85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc\" (UID: \"c47f0a31-b4dc-4648-88f4-3b54bf894a85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" Oct 30 16:48:44.813705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.813678 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c47f0a31-b4dc-4648-88f4-3b54bf894a85-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc\" (UID: \"c47f0a31-b4dc-4648-88f4-3b54bf894a85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" Oct 30 16:48:44.842502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.842427 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-httcj\" (UniqueName: \"kubernetes.io/projected/c47f0a31-b4dc-4648-88f4-3b54bf894a85-kube-api-access-httcj\") pod \"payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc\" (UID: \"c47f0a31-b4dc-4648-88f4-3b54bf894a85\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" Oct 30 16:48:44.967790 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:44.967649 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" Oct 30 16:48:45.086959 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:45.086922 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv"] Oct 30 16:48:45.094941 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:48:45.094909 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1d45e6_7bbe_4633_a4e9_c8aa8e135cdb.slice/crio-18aa70ba521d7935ad52c117947f1fcb4a71b9e905d9c69d9fa1f4e87d7f797c WatchSource:0}: Error finding container 18aa70ba521d7935ad52c117947f1fcb4a71b9e905d9c69d9fa1f4e87d7f797c: Status 404 returned error can't find the container with id 18aa70ba521d7935ad52c117947f1fcb4a71b9e905d9c69d9fa1f4e87d7f797c Oct 30 16:48:45.269148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:45.269109 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc"] Oct 30 16:48:45.272894 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:48:45.272857 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47f0a31_b4dc_4648_88f4_3b54bf894a85.slice/crio-09ee7d74edc7d0a44d8e4126eaf281aca6d581ea7e0538b87fc573a1c7e572e8 WatchSource:0}: Error finding container 09ee7d74edc7d0a44d8e4126eaf281aca6d581ea7e0538b87fc573a1c7e572e8: Status 404 returned error can't find the container with id 09ee7d74edc7d0a44d8e4126eaf281aca6d581ea7e0538b87fc573a1c7e572e8 Oct 30 16:48:45.417860 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:45.417804 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" event={"ID":"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb","Type":"ContainerStarted","Data":"18aa70ba521d7935ad52c117947f1fcb4a71b9e905d9c69d9fa1f4e87d7f797c"} Oct 30 16:48:45.419796 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:45.419769 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" event={"ID":"c47f0a31-b4dc-4648-88f4-3b54bf894a85","Type":"ContainerStarted","Data":"92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b"} Oct 30 16:48:45.419933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:45.419818 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" event={"ID":"c47f0a31-b4dc-4648-88f4-3b54bf894a85","Type":"ContainerStarted","Data":"09ee7d74edc7d0a44d8e4126eaf281aca6d581ea7e0538b87fc573a1c7e572e8"} Oct 30 16:48:45.419933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:45.419905 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" Oct 30 16:48:45.470413 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:45.470347 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" podStartSLOduration=1.470327231 podStartE2EDuration="1.470327231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:45.467436808 +0000 UTC m=+1687.216026233" watchObservedRunningTime="2026-10-30 16:48:45.470327231 +0000 UTC m=+1687.218916621" Oct 30 16:48:46.427524 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:46.427488 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" event={"ID":"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb","Type":"ContainerStarted","Data":"6c270500b158575e1880c2fd24fa205ed66ec962e1ba0bb0802b9711bb24d39c"} Oct 30 16:48:46.428008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:46.427815 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:46.468546 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:46.468495 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" podStartSLOduration=2.066328571 podStartE2EDuration="2.468479707s" totalImagesPullingTime="402.151136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:46.467945839 +0000 UTC m=+1688.216535226" watchObservedRunningTime="2026-10-30 16:48:46.468479707 +0000 UTC m=+1688.217069094" Oct 30 16:48:48.233994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:48.231228 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:48:49.032801 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:49.032759 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j"] Oct 30 16:48:49.045042 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:49.045002 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s4m2j"] Oct 30 16:48:50.394905 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:50.394865 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97eb915b-40dc-4ab9-bde8-3410417ca78f" path="/var/lib/kubelet/pods/97eb915b-40dc-4ab9-bde8-3410417ca78f/volumes" Oct 30 16:48:52.443077 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:52.443044 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:48:56.431236 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:56.431198 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" Oct 30 16:48:57.427224 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:57.427178 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:48:57.427452 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:57.427285 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:48:58.501212 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:58.501180 5838 generic.go:356] "Generic (PLEG): container finished" podID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerID="5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8" exitCode=0 Oct 30 16:48:58.501652 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:58.501231 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" event={"ID":"d95fe69c-92eb-4331-b46a-3cbab2f9a864","Type":"ContainerDied","Data":"5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8"} Oct 30 16:48:58.501652 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:58.501588 5838 scope.go:122] "RemoveContainer" containerID="5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8" Oct 30 16:48:58.502914 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:58.502859 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:48:58.503072 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:58.503000 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:48:58.503159 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:58.503131 5838 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 16:48:59.509002 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:48:59.508951 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" event={"ID":"d95fe69c-92eb-4331-b46a-3cbab2f9a864","Type":"ContainerStarted","Data":"aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888"} Oct 30 16:49:00.440400 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:00.440349 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:49:00.440691 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:00.440667 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 16:49:00.440787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:00.440719 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 16:49:01.442394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:01.442356 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 16:49:01.443056 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:01.443024 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 16:49:01.796497 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:49:01.796456 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:49:02.440330 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:02.440291 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 16:49:02.440632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:02.440601 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 16:49:02.840994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:02.840431 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:49:03.440549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:03.440510 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 16:49:03.440738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:03.440584 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 16:49:04.440441 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:04.440404 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 16:49:04.440862 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:04.440458 5838 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 16:49:05.443685 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:05.443653 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:49:05.444099 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:05.444019 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:49:05.444743 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:05.444725 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:49:17.715754 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:17.715711 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:49:30.935353 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:30.935301 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:49:31.797445 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:49:31.797403 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:49:40.909437 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:40.909400 5838 scope.go:122] "RemoveContainer" containerID="9a9dd273f44e04acd3ec82a9a0e90d6e27d4048f0047f7a0162cefede0ed0910" Oct 30 16:49:40.918787 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:40.918765 5838 scope.go:122] "RemoveContainer" containerID="da9f3216f716411ee60a4cc91243483793509b858b5c212495dbf49ff16a3f9f" Oct 30 16:49:40.927806 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:40.927781 5838 scope.go:122] "RemoveContainer" containerID="7b18bcc2919f2e3e7bb19cf8e788045cc8fa5052e94e52ac2df5580014dc101b" Oct 30 16:49:40.975090 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:40.975059 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc"] Oct 30 16:49:42.344451 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.344414 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv"] Oct 30 16:49:42.344866 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.344662 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" podUID="cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb" containerName="maas-api" containerID="cri-o://6c270500b158575e1880c2fd24fa205ed66ec962e1ba0bb0802b9711bb24d39c" gracePeriod=30 Oct 30 16:49:42.435756 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.435711 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" podUID="cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.165:8443/health\": dial tcp 10.134.0.165:8443: connect: connection refused" Oct 30 16:49:42.804123 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.804092 5838 generic.go:356] "Generic (PLEG): container finished" podID="cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb" containerID="6c270500b158575e1880c2fd24fa205ed66ec962e1ba0bb0802b9711bb24d39c" exitCode=0 Oct 30 16:49:42.804296 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.804167 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" event={"ID":"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb","Type":"ContainerDied","Data":"6c270500b158575e1880c2fd24fa205ed66ec962e1ba0bb0802b9711bb24d39c"} Oct 30 16:49:42.850634 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.850605 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:49:42.921879 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.921787 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-openshift-service-ca\") pod \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " Oct 30 16:49:42.921879 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.921849 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-kube-api-access-hg8r7\" (UniqueName: \"kubernetes.io/projected/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-kube-api-access-hg8r7\") pod \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " Oct 30 16:49:42.922174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.922050 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-maas-api-tls\") pod \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " Oct 30 16:49:42.922174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.922133 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-metrics-certs\") pod \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\" (UID: \"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb\") " Oct 30 16:49:42.922312 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.922280 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-openshift-service-ca" pod "cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb" (UID: "cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:42.922502 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.922480 5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-openshift-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:42.924603 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.924564 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-maas-api-tls" pod "cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb" (UID: "cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:49:42.924725 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.924656 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-metrics-certs" pod "cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb" (UID: "cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:49:42.924725 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:42.924682 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-kube-api-access-hg8r7" pod "cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb" (UID: "cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb"). InnerVolumeSpecName "kube-api-access-hg8r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:43.023540 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:43.023503 5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-metrics-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:43.023540 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:43.023537 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg8r7\" (UniqueName: \"kubernetes.io/projected/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-kube-api-access-hg8r7\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:43.023540 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:43.023548 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb-maas-api-tls\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:43.814749 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:43.814712 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" event={"ID":"cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb","Type":"ContainerDied","Data":"18aa70ba521d7935ad52c117947f1fcb4a71b9e905d9c69d9fa1f4e87d7f797c"} Oct 30 16:49:43.815201 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:43.814797 5838 scope.go:122] "RemoveContainer" containerID="6c270500b158575e1880c2fd24fa205ed66ec962e1ba0bb0802b9711bb24d39c" Oct 30 16:49:43.815201 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:43.814726 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv" Oct 30 16:49:43.863875 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:43.863841 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv"] Oct 30 16:49:43.875049 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:43.875021 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3060bb-7f9df9d5c6-5qpfv"] Oct 30 16:49:44.394598 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:44.394563 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb" path="/var/lib/kubelet/pods/cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb/volumes" Oct 30 16:49:44.755991 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:44.748613 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:49:45.976136 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:45.976082 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" podUID="c47f0a31-b4dc-4648-88f4-3b54bf894a85" containerName="payload-pre-processing" containerID="cri-o://92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b" gracePeriod=25 Oct 30 16:49:46.429539 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:46.429498 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" podUID="c47f0a31-b4dc-4648-88f4-3b54bf894a85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:49:50.041373 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:50.041338 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b"] Oct 30 16:49:50.041882 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:50.041590 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" containerID="cri-o://aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888" gracePeriod=30 Oct 30 16:49:50.447281 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:50.447200 5838 patch_prober.go:29] interesting pod/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:49:50.447281 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:50.447264 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:49:54.397080 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.397043 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n"] Oct 30 16:49:54.397706 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.397404 5838 cpu_manager.go:414] "containerMap: removing container" podUID="cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb" containerName="maas-api" Oct 30 16:49:54.397706 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.397414 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb" containerName="maas-api" Oct 30 16:49:54.397706 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.397506 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1d45e6-7bbe-4633-a4e9-c8aa8e135cdb" containerName="maas-api" Oct 30 16:49:54.402015 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.401993 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.428000 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.427958 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0ab40f41-kserve-self-signed-certs\"" Oct 30 16:49:54.431525 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.431498 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n"] Oct 30 16:49:54.546348 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.546316 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tls-certs\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.546540 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.546413 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-home\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.546540 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.546453 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tmp-dir\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.546663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.546547 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-model-cache\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.546663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.546580 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-dshm\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.546663 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.546602 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcmb\" (UniqueName: \"kubernetes.io/projected/a19bc095-b22a-44b3-8aa4-e6d4f988f946-kube-api-access-bjcmb\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.647809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.647705 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tls-certs\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.647809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.647773 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-home\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.648068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.647891 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tmp-dir\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.648068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.647964 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-model-cache\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.648068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.648026 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-dshm\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.648068 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.648042 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcmb\" (UniqueName: \"kubernetes.io/projected/a19bc095-b22a-44b3-8aa4-e6d4f988f946-kube-api-access-bjcmb\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.648303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.648175 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-home\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.648303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.648260 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tmp-dir\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.648303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.648286 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-model-cache\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.650230 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.650204 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-dshm\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.650460 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.650440 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tls-certs\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.670869 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.670844 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcmb\" (UniqueName: \"kubernetes.io/projected/a19bc095-b22a-44b3-8aa4-e6d4f988f946-kube-api-access-bjcmb\") pod \"e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:54.715768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:54.715732 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:55.017632 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.017590 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n"] Oct 30 16:49:55.021273 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:49:55.021240 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda19bc095_b22a_44b3_8aa4_e6d4f988f946.slice/crio-a30eb64959dc84df23983528e99b4191dc4ffad867208ba52c7abbf92987619e WatchSource:0}: Error finding container a30eb64959dc84df23983528e99b4191dc4ffad867208ba52c7abbf92987619e: Status 404 returned error can't find the container with id a30eb64959dc84df23983528e99b4191dc4ffad867208ba52c7abbf92987619e Oct 30 16:49:55.291567 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.291540 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:49:55.456884 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.456848 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-envoy\") pod \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " Oct 30 16:49:55.456884 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.456892 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-certs\") pod \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " Oct 30 16:49:55.457457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.456927 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-token\" (UniqueName: \"kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-token\") pod \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " Oct 30 16:49:55.457457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.456947 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istiod-ca-cert\") pod \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " Oct 30 16:49:55.457457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457055 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-socket\") pod \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " Oct 30 16:49:55.457457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457100 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-kube-api-access-s79nz\" (UniqueName: \"kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-kube-api-access-s79nz\") pod \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " Oct 30 16:49:55.457457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457230 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-certs" pod "d95fe69c-92eb-4331-b46a-3cbab2f9a864" (UID: "d95fe69c-92eb-4331-b46a-3cbab2f9a864"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:55.457457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457238 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-podinfo\") pod \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " Oct 30 16:49:55.457457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457364 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-credential-socket\") pod \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " Oct 30 16:49:55.457457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457399 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-data\") pod \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\" (UID: \"d95fe69c-92eb-4331-b46a-3cbab2f9a864\") " Oct 30 16:49:55.457909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457459 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-socket" pod "d95fe69c-92eb-4331-b46a-3cbab2f9a864" (UID: "d95fe69c-92eb-4331-b46a-3cbab2f9a864"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:55.457909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457602 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-credential-socket" pod "d95fe69c-92eb-4331-b46a-3cbab2f9a864" (UID: "d95fe69c-92eb-4331-b46a-3cbab2f9a864"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:55.457909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457667 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-data" pod "d95fe69c-92eb-4331-b46a-3cbab2f9a864" (UID: "d95fe69c-92eb-4331-b46a-3cbab2f9a864"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:55.457909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457778 5838 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:55.457909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457790 5838 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-credential-socket\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:55.457909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457799 5838 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-data\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:55.457909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457807 5838 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-workload-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:55.457909 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.457883 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istiod-ca-cert" pod "d95fe69c-92eb-4331-b46a-3cbab2f9a864" (UID: "d95fe69c-92eb-4331-b46a-3cbab2f9a864"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:55.459280 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.459251 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-token" pod "d95fe69c-92eb-4331-b46a-3cbab2f9a864" (UID: "d95fe69c-92eb-4331-b46a-3cbab2f9a864"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:55.459280 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.459268 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-envoy" pod "d95fe69c-92eb-4331-b46a-3cbab2f9a864" (UID: "d95fe69c-92eb-4331-b46a-3cbab2f9a864"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:55.459452 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.459434 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-podinfo" pod "d95fe69c-92eb-4331-b46a-3cbab2f9a864" (UID: "d95fe69c-92eb-4331-b46a-3cbab2f9a864"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:49:55.459544 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.459527 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-kube-api-access-s79nz" pod "d95fe69c-92eb-4331-b46a-3cbab2f9a864" (UID: "d95fe69c-92eb-4331-b46a-3cbab2f9a864"). InnerVolumeSpecName "kube-api-access-s79nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:55.558549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.558453 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s79nz\" (UniqueName: \"kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-kube-api-access-s79nz\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:55.558549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.558486 5838 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-podinfo\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:55.558549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.558496 5838 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-envoy\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:55.558549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.558505 5838 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istio-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:55.558549 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.558517 5838 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95fe69c-92eb-4331-b46a-3cbab2f9a864-istiod-ca-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:49:55.883559 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.883472 5838 generic.go:356] "Generic (PLEG): container finished" podID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerID="aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888" exitCode=0 Oct 30 16:49:55.883559 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.883530 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" Oct 30 16:49:55.883795 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.883558 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" event={"ID":"d95fe69c-92eb-4331-b46a-3cbab2f9a864","Type":"ContainerDied","Data":"aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888"} Oct 30 16:49:55.883795 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.883616 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b" event={"ID":"d95fe69c-92eb-4331-b46a-3cbab2f9a864","Type":"ContainerDied","Data":"f6ea3d4502399b19319a616ab87e0329718c9873168720d651bb742f14be87b3"} Oct 30 16:49:55.883795 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.883639 5838 scope.go:122] "RemoveContainer" containerID="aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888" Oct 30 16:49:55.885320 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.885290 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" event={"ID":"a19bc095-b22a-44b3-8aa4-e6d4f988f946","Type":"ContainerStarted","Data":"666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91"} Oct 30 16:49:55.885457 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.885327 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" event={"ID":"a19bc095-b22a-44b3-8aa4-e6d4f988f946","Type":"ContainerStarted","Data":"a30eb64959dc84df23983528e99b4191dc4ffad867208ba52c7abbf92987619e"} Oct 30 16:49:55.893333 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.893317 5838 scope.go:122] "RemoveContainer" containerID="5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8" Oct 30 16:49:55.902233 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.902210 5838 scope.go:122] "RemoveContainer" containerID="aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888" Oct 30 16:49:55.902471 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:49:55.902455 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888\": container with ID starting with aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888 not found: ID does not exist" containerID="aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888" Oct 30 16:49:55.902522 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.902479 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888"} err="failed to get container status \"aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888\": rpc error: code = NotFound desc = could not find container \"aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888\": container with ID starting with aea44d4811ce8ea0f940df64ef19b07ea1df9158f3c82a8cd5da9a19a4da6888 not found: ID does not exist" Oct 30 16:49:55.902522 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.902494 5838 scope.go:122] "RemoveContainer" containerID="5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8" Oct 30 16:49:55.902739 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:49:55.902722 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8\": container with ID starting with 5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8 not found: ID does not exist" containerID="5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8" Oct 30 16:49:55.902779 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.902746 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8"} err="failed to get container status \"5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8\": rpc error: code = NotFound desc = could not find container \"5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8\": container with ID starting with 5ccbd6c3e33a7d6d331d8bcc7ee213de4601c8f03201b980a7a5bae325efe4a8 not found: ID does not exist" Oct 30 16:49:55.939493 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.939440 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" podStartSLOduration=1.939427195 podStartE2EDuration="1.939427195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:55.937532739 +0000 UTC m=+1757.686122127" watchObservedRunningTime="2026-10-30 16:49:55.939427195 +0000 UTC m=+1757.688016581" Oct 30 16:49:55.973003 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.972706 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b"] Oct 30 16:49:55.983965 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:55.983937 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3060bb-openshift-default-77b6d86845dqq9b"] Oct 30 16:49:56.395901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:56.395856 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" path="/var/lib/kubelet/pods/d95fe69c-92eb-4331-b46a-3cbab2f9a864/volumes" Oct 30 16:49:56.429484 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:56.429437 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" podUID="c47f0a31-b4dc-4648-88f4-3b54bf894a85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:49:56.716275 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:56.716188 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:49:56.848467 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:49:56.848425 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:50:01.796536 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:50:01.796495 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:50:04.716311 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:04.716268 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:50:04.729579 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:04.729553 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:50:04.956076 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:04.956046 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:50:06.220874 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.220848 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" Oct 30 16:50:06.364328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.364295 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c47f0a31-b4dc-4648-88f4-3b54bf894a85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c47f0a31-b4dc-4648-88f4-3b54bf894a85-plugins-config-volume\") pod \"c47f0a31-b4dc-4648-88f4-3b54bf894a85\" (UID: \"c47f0a31-b4dc-4648-88f4-3b54bf894a85\") " Oct 30 16:50:06.364553 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.364370 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c47f0a31-b4dc-4648-88f4-3b54bf894a85-kube-api-access-httcj\" (UniqueName: \"kubernetes.io/projected/c47f0a31-b4dc-4648-88f4-3b54bf894a85-kube-api-access-httcj\") pod \"c47f0a31-b4dc-4648-88f4-3b54bf894a85\" (UID: \"c47f0a31-b4dc-4648-88f4-3b54bf894a85\") " Oct 30 16:50:06.364752 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.364726 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c47f0a31-b4dc-4648-88f4-3b54bf894a85-plugins-config-volume" pod "c47f0a31-b4dc-4648-88f4-3b54bf894a85" (UID: "c47f0a31-b4dc-4648-88f4-3b54bf894a85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:06.366467 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.366447 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47f0a31-b4dc-4648-88f4-3b54bf894a85-kube-api-access-httcj" pod "c47f0a31-b4dc-4648-88f4-3b54bf894a85" (UID: "c47f0a31-b4dc-4648-88f4-3b54bf894a85"). InnerVolumeSpecName "kube-api-access-httcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:06.465394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.465350 5838 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c47f0a31-b4dc-4648-88f4-3b54bf894a85-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:50:06.465394 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.465394 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-httcj\" (UniqueName: \"kubernetes.io/projected/c47f0a31-b4dc-4648-88f4-3b54bf894a85-kube-api-access-httcj\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:50:06.958241 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.958205 5838 generic.go:356] "Generic (PLEG): container finished" podID="c47f0a31-b4dc-4648-88f4-3b54bf894a85" containerID="92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b" exitCode=0 Oct 30 16:50:06.958421 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.958250 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" Oct 30 16:50:06.958421 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.958289 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" event={"ID":"c47f0a31-b4dc-4648-88f4-3b54bf894a85","Type":"ContainerDied","Data":"92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b"} Oct 30 16:50:06.958421 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.958340 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc" event={"ID":"c47f0a31-b4dc-4648-88f4-3b54bf894a85","Type":"ContainerDied","Data":"09ee7d74edc7d0a44d8e4126eaf281aca6d581ea7e0538b87fc573a1c7e572e8"} Oct 30 16:50:06.958421 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.958363 5838 scope.go:122] "RemoveContainer" containerID="92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b" Oct 30 16:50:06.967303 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.967281 5838 scope.go:122] "RemoveContainer" containerID="92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b" Oct 30 16:50:06.967593 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:50:06.967574 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b\": container with ID starting with 92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b not found: ID does not exist" containerID="92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b" Oct 30 16:50:06.967655 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:06.967603 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b"} err="failed to get container status \"92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b\": rpc error: code = NotFound desc = could not find container \"92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b\": container with ID starting with 92d9b18fa3e2b752f44bd92ae50ba14c86956ed41c385cad088fdfd8561b929b not found: ID does not exist" Oct 30 16:50:07.010847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:07.010813 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc"] Oct 30 16:50:07.021416 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:07.021385 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3060bb-64585qp2jc"] Oct 30 16:50:08.394432 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:08.394403 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c47f0a31-b4dc-4648-88f4-3b54bf894a85" path="/var/lib/kubelet/pods/c47f0a31-b4dc-4648-88f4-3b54bf894a85/volumes" Oct 30 16:50:16.382199 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.382161 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2"] Oct 30 16:50:16.382780 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.382758 5838 cpu_manager.go:414] "containerMap: removing container" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" Oct 30 16:50:16.382780 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.382781 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" Oct 30 16:50:16.382910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.382795 5838 cpu_manager.go:414] "containerMap: removing container" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" Oct 30 16:50:16.382910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.382801 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" Oct 30 16:50:16.382910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.382824 5838 cpu_manager.go:414] "containerMap: removing container" podUID="c47f0a31-b4dc-4648-88f4-3b54bf894a85" containerName="payload-pre-processing" Oct 30 16:50:16.382910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.382830 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c47f0a31-b4dc-4648-88f4-3b54bf894a85" containerName="payload-pre-processing" Oct 30 16:50:16.382910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.382892 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" Oct 30 16:50:16.382910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.382904 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="c47f0a31-b4dc-4648-88f4-3b54bf894a85" containerName="payload-pre-processing" Oct 30 16:50:16.387960 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.387919 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.406551 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.406507 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2"] Oct 30 16:50:16.414464 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.414428 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0ab40f41-kserve-self-signed-certs\"" Oct 30 16:50:16.561019 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.560939 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tls-certs\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.561231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.561119 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-model-cache\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.561305 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.561275 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc4sz\" (UniqueName: \"kubernetes.io/projected/1983d9b4-c329-4817-b2ff-960d8f42c6f5-kube-api-access-sc4sz\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.561358 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.561320 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tmp-dir\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.561479 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.561451 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-home\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.561625 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.561506 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-dshm\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.662544 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.662455 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-model-cache\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.662544 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.662518 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc4sz\" (UniqueName: \"kubernetes.io/projected/1983d9b4-c329-4817-b2ff-960d8f42c6f5-kube-api-access-sc4sz\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.662544 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.662540 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tmp-dir\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.662836 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.662569 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-home\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.662836 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.662594 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-dshm\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.662836 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.662629 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tls-certs\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.662961 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.662932 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-model-cache\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.663029 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.662984 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-home\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.663066 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.663046 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tmp-dir\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.664933 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.664909 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-dshm\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.665098 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.665081 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tls-certs\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.687030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.686997 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc4sz\" (UniqueName: \"kubernetes.io/projected/1983d9b4-c329-4817-b2ff-960d8f42c6f5-kube-api-access-sc4sz\") pod \"e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:16.702898 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:16.702861 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:17.003301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:17.003247 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2"] Oct 30 16:50:17.007708 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:50:17.007676 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1983d9b4_c329_4817_b2ff_960d8f42c6f5.slice/crio-b89add359c555899756195d8af538ade4df246511a0c779b72203eac881f3832 WatchSource:0}: Error finding container b89add359c555899756195d8af538ade4df246511a0c779b72203eac881f3832: Status 404 returned error can't find the container with id b89add359c555899756195d8af538ade4df246511a0c779b72203eac881f3832 Oct 30 16:50:17.014901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:17.014871 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" event={"ID":"1983d9b4-c329-4817-b2ff-960d8f42c6f5","Type":"ContainerStarted","Data":"b89add359c555899756195d8af538ade4df246511a0c779b72203eac881f3832"} Oct 30 16:50:18.022151 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:18.022111 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" event={"ID":"1983d9b4-c329-4817-b2ff-960d8f42c6f5","Type":"ContainerStarted","Data":"e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004"} Oct 30 16:50:18.063181 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:18.062912 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" podStartSLOduration=2.062895074 podStartE2EDuration="2.062895074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:18.061144602 +0000 UTC m=+1779.809733989" watchObservedRunningTime="2026-10-30 16:50:18.062895074 +0000 UTC m=+1779.811484465" Oct 30 16:50:18.703176 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:18.703129 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:18.708925 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:18.708895 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:50:26.703958 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:26.703921 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:26.716611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:26.716583 5838 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:27.126514 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:27.126481 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:50:31.797431 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:50:31.797388 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:50:34.725588 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:34.725548 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:50:35.415828 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.415786 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs"] Oct 30 16:50:35.416587 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.416560 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95fe69c-92eb-4331-b46a-3cbab2f9a864" containerName="istio-proxy" Oct 30 16:50:35.420295 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.420266 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:35.438427 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.438400 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs"] Oct 30 16:50:35.446366 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.446330 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:50:35.446509 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.446367 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:50:35.483869 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.483835 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:50:35.483869 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.483861 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:50:35.488008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.487941 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4djbs\" (UID: \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:35.488161 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.488029 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58hf\" (UniqueName: \"kubernetes.io/projected/6342b4fe-4d83-4dc6-94ac-25cb432ae497-kube-api-access-d58hf\") pod \"maas-api-revoke-sub-452b6d7b-4djbs\" (UID: \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:35.488161 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.488138 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4djbs\" (UID: \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:35.589404 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.589344 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4djbs\" (UID: \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:35.589629 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.589475 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4djbs\" (UID: \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:35.589629 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.589510 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58hf\" (UniqueName: \"kubernetes.io/projected/6342b4fe-4d83-4dc6-94ac-25cb432ae497-kube-api-access-d58hf\") pod \"maas-api-revoke-sub-452b6d7b-4djbs\" (UID: \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:35.590231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.590198 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4djbs\" (UID: \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:35.592224 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.592195 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4djbs\" (UID: \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:35.612650 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.612613 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58hf\" (UniqueName: \"kubernetes.io/projected/6342b4fe-4d83-4dc6-94ac-25cb432ae497-kube-api-access-d58hf\") pod \"maas-api-revoke-sub-452b6d7b-4djbs\" (UID: \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:35.739525 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:35.739477 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:36.057472 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:36.057385 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs"] Oct 30 16:50:36.062943 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:50:36.062914 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6342b4fe_4d83_4dc6_94ac_25cb432ae497.slice/crio-b69301f12a9bbc2132c59fe0c36fa40ed54513ada85c3c11e1ea95702990c71f WatchSource:0}: Error finding container b69301f12a9bbc2132c59fe0c36fa40ed54513ada85c3c11e1ea95702990c71f: Status 404 returned error can't find the container with id b69301f12a9bbc2132c59fe0c36fa40ed54513ada85c3c11e1ea95702990c71f Oct 30 16:50:36.165204 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:36.165166 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" event={"ID":"6342b4fe-4d83-4dc6-94ac-25cb432ae497","Type":"ContainerStarted","Data":"b69301f12a9bbc2132c59fe0c36fa40ed54513ada85c3c11e1ea95702990c71f"} Oct 30 16:50:37.185328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:37.184644 5838 generic.go:356] "Generic (PLEG): container finished" podID="6342b4fe-4d83-4dc6-94ac-25cb432ae497" containerID="e4cc2bc2375771b6116ebeb25e4ee02cf1012987716610c5be0df9cd4bd86d81" exitCode=0 Oct 30 16:50:37.185328 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:37.184811 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" event={"ID":"6342b4fe-4d83-4dc6-94ac-25cb432ae497","Type":"ContainerDied","Data":"e4cc2bc2375771b6116ebeb25e4ee02cf1012987716610c5be0df9cd4bd86d81"} Oct 30 16:50:38.432473 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:38.432449 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:38.525511 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:38.525470 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-xks-ca\") pod \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\" (UID: \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\") " Oct 30 16:50:38.526531 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:38.525864 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-openshift-ca\") pod \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\" (UID: \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\") " Oct 30 16:50:38.526784 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:38.526760 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6342b4fe-4d83-4dc6-94ac-25cb432ae497-kube-api-access-d58hf\" (UniqueName: \"kubernetes.io/projected/6342b4fe-4d83-4dc6-94ac-25cb432ae497-kube-api-access-d58hf\") pod \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\" (UID: \"6342b4fe-4d83-4dc6-94ac-25cb432ae497\") " Oct 30 16:50:38.530186 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:38.530158 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6342b4fe-4d83-4dc6-94ac-25cb432ae497-kube-api-access-d58hf" pod "6342b4fe-4d83-4dc6-94ac-25cb432ae497" (UID: "6342b4fe-4d83-4dc6-94ac-25cb432ae497"). InnerVolumeSpecName "kube-api-access-d58hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:38.534156 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:38.534123 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-openshift-ca" pod "6342b4fe-4d83-4dc6-94ac-25cb432ae497" (UID: "6342b4fe-4d83-4dc6-94ac-25cb432ae497"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:38.540722 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:38.540696 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-xks-ca" pod "6342b4fe-4d83-4dc6-94ac-25cb432ae497" (UID: "6342b4fe-4d83-4dc6-94ac-25cb432ae497"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:38.628254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:38.628214 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:50:38.628254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:38.628247 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58hf\" (UniqueName: \"kubernetes.io/projected/6342b4fe-4d83-4dc6-94ac-25cb432ae497-kube-api-access-d58hf\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:50:38.628254 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:38.628259 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6342b4fe-4d83-4dc6-94ac-25cb432ae497-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:50:38.661620 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:38.661591 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:50:38.671206 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:38.671182 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v4lzf_4b13dfe3-9da5-4be8-9051-b1bbff1b083b/ovn-acl-logging/0.log" Oct 30 16:50:39.198982 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:39.198933 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" event={"ID":"6342b4fe-4d83-4dc6-94ac-25cb432ae497","Type":"ContainerDied","Data":"b69301f12a9bbc2132c59fe0c36fa40ed54513ada85c3c11e1ea95702990c71f"} Oct 30 16:50:39.198982 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:39.198948 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4djbs" Oct 30 16:50:39.199194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:39.199029 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b69301f12a9bbc2132c59fe0c36fa40ed54513ada85c3c11e1ea95702990c71f" Oct 30 16:50:40.958668 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:40.958629 5838 scope.go:122] "RemoveContainer" containerID="fc0ee3c75d8c727f6f3bf3ad68d120fbbff85fab113b2c2dce7eaaf3a6e476ce" Oct 30 16:50:40.967845 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:40.967825 5838 scope.go:122] "RemoveContainer" containerID="b385730b4a5b1c4fa48365c629e97b9902e2a45d8c7dacb8bea19cd13a57f9d3" Oct 30 16:50:40.976070 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:40.976042 5838 scope.go:122] "RemoveContainer" containerID="7bb783153962ffd4618c97f2b8a30cf5c790a98f814666a7a378dc1ec65058f4" Oct 30 16:50:46.578240 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:46.578192 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n"] Oct 30 16:50:48.914210 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:48.914170 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:50:50.894154 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:50.893928 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2"] Oct 30 16:50:54.603203 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.603159 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4"] Oct 30 16:50:54.603824 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.603800 5838 cpu_manager.go:414] "containerMap: removing container" podUID="6342b4fe-4d83-4dc6-94ac-25cb432ae497" containerName="revoke-keys" Oct 30 16:50:54.603876 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.603827 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6342b4fe-4d83-4dc6-94ac-25cb432ae497" containerName="revoke-keys" Oct 30 16:50:54.604008 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.603994 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="6342b4fe-4d83-4dc6-94ac-25cb432ae497" containerName="revoke-keys" Oct 30 16:50:54.609731 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.609709 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:54.626611 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.626580 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4"] Oct 30 16:50:54.633797 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.633751 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:50:54.634009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.633815 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:50:54.652636 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.652602 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:50:54.652770 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.652653 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:50:54.774619 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.774582 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-skhd4\" (UID: \"0487b0d7-2a8b-4267-862c-e85749810588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:54.774804 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.774651 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw8zm\" (UniqueName: \"kubernetes.io/projected/0487b0d7-2a8b-4267-862c-e85749810588-kube-api-access-dw8zm\") pod \"maas-api-revoke-sub-3e31a506-skhd4\" (UID: \"0487b0d7-2a8b-4267-862c-e85749810588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:54.774903 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.774874 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-skhd4\" (UID: \"0487b0d7-2a8b-4267-862c-e85749810588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:54.887857 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.887755 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-skhd4\" (UID: \"0487b0d7-2a8b-4267-862c-e85749810588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:54.888177 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.888147 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw8zm\" (UniqueName: \"kubernetes.io/projected/0487b0d7-2a8b-4267-862c-e85749810588-kube-api-access-dw8zm\") pod \"maas-api-revoke-sub-3e31a506-skhd4\" (UID: \"0487b0d7-2a8b-4267-862c-e85749810588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:54.888420 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.888398 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-skhd4\" (UID: \"0487b0d7-2a8b-4267-862c-e85749810588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:54.895197 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.895165 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-skhd4\" (UID: \"0487b0d7-2a8b-4267-862c-e85749810588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:54.900579 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.899505 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-skhd4\" (UID: \"0487b0d7-2a8b-4267-862c-e85749810588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:54.911737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.911713 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw8zm\" (UniqueName: \"kubernetes.io/projected/0487b0d7-2a8b-4267-862c-e85749810588-kube-api-access-dw8zm\") pod \"maas-api-revoke-sub-3e31a506-skhd4\" (UID: \"0487b0d7-2a8b-4267-862c-e85749810588\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:54.929383 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:54.929129 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:55.377004 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:55.373180 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4"] Oct 30 16:50:56.185053 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:56.181493 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:50:56.343011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:56.340508 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" event={"ID":"0487b0d7-2a8b-4267-862c-e85749810588","Type":"ContainerStarted","Data":"e0434443cb38f1bba35b12e7d559fe57cc57f826cec35fe50b4abadb1eddff18"} Oct 30 16:50:56.343011 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:56.340581 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" event={"ID":"0487b0d7-2a8b-4267-862c-e85749810588","Type":"ContainerStarted","Data":"b8aee9468a5a56ac00c4c05ae3d22f6a7e8d4d54d792f3ddf1c2fbef96e04f5d"} Oct 30 16:50:57.348012 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:57.347960 5838 generic.go:356] "Generic (PLEG): container finished" podID="0487b0d7-2a8b-4267-862c-e85749810588" containerID="e0434443cb38f1bba35b12e7d559fe57cc57f826cec35fe50b4abadb1eddff18" exitCode=0 Oct 30 16:50:57.348423 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:57.348012 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" event={"ID":"0487b0d7-2a8b-4267-862c-e85749810588","Type":"ContainerDied","Data":"e0434443cb38f1bba35b12e7d559fe57cc57f826cec35fe50b4abadb1eddff18"} Oct 30 16:50:58.498346 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:58.498314 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:58.629524 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:58.629437 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-openshift-ca\") pod \"0487b0d7-2a8b-4267-862c-e85749810588\" (UID: \"0487b0d7-2a8b-4267-862c-e85749810588\") " Oct 30 16:50:58.629524 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:58.629505 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0487b0d7-2a8b-4267-862c-e85749810588-kube-api-access-dw8zm\" (UniqueName: \"kubernetes.io/projected/0487b0d7-2a8b-4267-862c-e85749810588-kube-api-access-dw8zm\") pod \"0487b0d7-2a8b-4267-862c-e85749810588\" (UID: \"0487b0d7-2a8b-4267-862c-e85749810588\") " Oct 30 16:50:58.629732 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:58.629616 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-xks-ca\") pod \"0487b0d7-2a8b-4267-862c-e85749810588\" (UID: \"0487b0d7-2a8b-4267-862c-e85749810588\") " Oct 30 16:50:58.629886 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:58.629863 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-openshift-ca" pod "0487b0d7-2a8b-4267-862c-e85749810588" (UID: "0487b0d7-2a8b-4267-862c-e85749810588"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:58.630020 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:58.630005 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:50:58.631738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:58.631709 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-xks-ca" pod "0487b0d7-2a8b-4267-862c-e85749810588" (UID: "0487b0d7-2a8b-4267-862c-e85749810588"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:58.631738 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:58.631715 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0487b0d7-2a8b-4267-862c-e85749810588-kube-api-access-dw8zm" pod "0487b0d7-2a8b-4267-862c-e85749810588" (UID: "0487b0d7-2a8b-4267-862c-e85749810588"). InnerVolumeSpecName "kube-api-access-dw8zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:58.730697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:58.730645 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw8zm\" (UniqueName: \"kubernetes.io/projected/0487b0d7-2a8b-4267-862c-e85749810588-kube-api-access-dw8zm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:50:58.730697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:58.730688 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0487b0d7-2a8b-4267-862c-e85749810588-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:50:59.361857 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:59.361821 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" event={"ID":"0487b0d7-2a8b-4267-862c-e85749810588","Type":"ContainerDied","Data":"b8aee9468a5a56ac00c4c05ae3d22f6a7e8d4d54d792f3ddf1c2fbef96e04f5d"} Oct 30 16:50:59.361857 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:59.361844 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-skhd4" Oct 30 16:50:59.362187 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:50:59.361876 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8aee9468a5a56ac00c4c05ae3d22f6a7e8d4d54d792f3ddf1c2fbef96e04f5d" Oct 30 16:51:01.599183 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.599117 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" podUID="a19bc095-b22a-44b3-8aa4-e6d4f988f946" containerName="main" containerID="cri-o://666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91" gracePeriod=45 Oct 30 16:51:01.805219 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:51:01.805176 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:51:01.850863 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.850799 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:51:01.859792 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.859767 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tls-certs\" (UniqueName: \"kubernetes.io/secret/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tls-certs\") pod \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " Oct 30 16:51:01.859912 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.859815 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-model-cache\") pod \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " Oct 30 16:51:01.859912 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.859859 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-home\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-home\") pod \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " Oct 30 16:51:01.859912 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.859880 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19bc095-b22a-44b3-8aa4-e6d4f988f946-kube-api-access-bjcmb\" (UniqueName: \"kubernetes.io/projected/a19bc095-b22a-44b3-8aa4-e6d4f988f946-kube-api-access-bjcmb\") pod \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " Oct 30 16:51:01.859912 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.859898 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tmp-dir\") pod \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " Oct 30 16:51:01.860170 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.859917 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-dshm\") pod \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\" (UID: \"a19bc095-b22a-44b3-8aa4-e6d4f988f946\") " Oct 30 16:51:01.860170 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.860149 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-model-cache" pod "a19bc095-b22a-44b3-8aa4-e6d4f988f946" (UID: "a19bc095-b22a-44b3-8aa4-e6d4f988f946"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:01.860261 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.860168 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tmp-dir" pod "a19bc095-b22a-44b3-8aa4-e6d4f988f946" (UID: "a19bc095-b22a-44b3-8aa4-e6d4f988f946"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:01.860261 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.860176 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-home" pod "a19bc095-b22a-44b3-8aa4-e6d4f988f946" (UID: "a19bc095-b22a-44b3-8aa4-e6d4f988f946"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:01.860261 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.860250 5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-model-cache\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:01.860365 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.860267 5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tmp-dir\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:01.862092 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.862063 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19bc095-b22a-44b3-8aa4-e6d4f988f946-kube-api-access-bjcmb" pod "a19bc095-b22a-44b3-8aa4-e6d4f988f946" (UID: "a19bc095-b22a-44b3-8aa4-e6d4f988f946"). InnerVolumeSpecName "kube-api-access-bjcmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:01.862216 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.862087 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-dshm" pod "a19bc095-b22a-44b3-8aa4-e6d4f988f946" (UID: "a19bc095-b22a-44b3-8aa4-e6d4f988f946"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:01.862216 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.862105 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tls-certs" pod "a19bc095-b22a-44b3-8aa4-e6d4f988f946" (UID: "a19bc095-b22a-44b3-8aa4-e6d4f988f946"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:01.961052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.961014 5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-home\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:01.961052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.961046 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjcmb\" (UniqueName: \"kubernetes.io/projected/a19bc095-b22a-44b3-8aa4-e6d4f988f946-kube-api-access-bjcmb\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:01.961052 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.961058 5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19bc095-b22a-44b3-8aa4-e6d4f988f946-dshm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:01.961301 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:01.961066 5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19bc095-b22a-44b3-8aa4-e6d4f988f946-tls-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:02.381177 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:02.381142 5838 generic.go:356] "Generic (PLEG): container finished" podID="a19bc095-b22a-44b3-8aa4-e6d4f988f946" containerID="666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91" exitCode=0 Oct 30 16:51:02.381366 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:02.381187 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" Oct 30 16:51:02.381366 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:02.381229 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" event={"ID":"a19bc095-b22a-44b3-8aa4-e6d4f988f946","Type":"ContainerDied","Data":"666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91"} Oct 30 16:51:02.381366 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:02.381279 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n" event={"ID":"a19bc095-b22a-44b3-8aa4-e6d4f988f946","Type":"ContainerDied","Data":"a30eb64959dc84df23983528e99b4191dc4ffad867208ba52c7abbf92987619e"} Oct 30 16:51:02.381366 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:02.381300 5838 scope.go:122] "RemoveContainer" containerID="666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91" Oct 30 16:51:02.391116 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:02.391098 5838 scope.go:122] "RemoveContainer" containerID="666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91" Oct 30 16:51:02.391435 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:51:02.391413 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91\": container with ID starting with 666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91 not found: ID does not exist" containerID="666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91" Oct 30 16:51:02.391535 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:02.391441 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91"} err="failed to get container status \"666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91\": rpc error: code = NotFound desc = could not find container \"666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91\": container with ID starting with 666a4d3d29ba5e580b17dde48a72eec8ed467ebed136f0b727d01117189ebe91 not found: ID does not exist" Oct 30 16:51:02.430652 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:02.430613 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n"] Oct 30 16:51:02.441854 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:02.441794 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0ab40f41-kserve-755587b969-qq25n"] Oct 30 16:51:04.395384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:04.395345 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19bc095-b22a-44b3-8aa4-e6d4f988f946" path="/var/lib/kubelet/pods/a19bc095-b22a-44b3-8aa4-e6d4f988f946/volumes" Oct 30 16:51:06.015758 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.015714 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:51:06.101624 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.101575 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" podUID="1983d9b4-c329-4817-b2ff-960d8f42c6f5" containerName="main" containerID="cri-o://e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004" gracePeriod=45 Oct 30 16:51:06.113547 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.113514 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" podUID="1983d9b4-c329-4817-b2ff-960d8f42c6f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": dial tcp 10.134.0.168:8000: connect: connection refused" Oct 30 16:51:06.370017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.369985 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:51:06.422115 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.422071 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tls-certs\") pod \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " Oct 30 16:51:06.422435 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.422410 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-dshm\") pod \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " Oct 30 16:51:06.422642 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.422622 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tmp-dir\") pod \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " Oct 30 16:51:06.422802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.422785 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-model-cache\") pod \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " Oct 30 16:51:06.423033 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.423014 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1983d9b4-c329-4817-b2ff-960d8f42c6f5-kube-api-access-sc4sz\" (UniqueName: \"kubernetes.io/projected/1983d9b4-c329-4817-b2ff-960d8f42c6f5-kube-api-access-sc4sz\") pod \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " Oct 30 16:51:06.426163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.426136 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-home\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-home\") pod \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\" (UID: \"1983d9b4-c329-4817-b2ff-960d8f42c6f5\") " Oct 30 16:51:06.430737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.429207 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tmp-dir" pod "1983d9b4-c329-4817-b2ff-960d8f42c6f5" (UID: "1983d9b4-c329-4817-b2ff-960d8f42c6f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:06.430737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.429513 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-home" pod "1983d9b4-c329-4817-b2ff-960d8f42c6f5" (UID: "1983d9b4-c329-4817-b2ff-960d8f42c6f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:06.430737 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.429883 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-model-cache" pod "1983d9b4-c329-4817-b2ff-960d8f42c6f5" (UID: "1983d9b4-c329-4817-b2ff-960d8f42c6f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:06.438207 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.438154 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1983d9b4-c329-4817-b2ff-960d8f42c6f5-kube-api-access-sc4sz" pod "1983d9b4-c329-4817-b2ff-960d8f42c6f5" (UID: "1983d9b4-c329-4817-b2ff-960d8f42c6f5"). InnerVolumeSpecName "kube-api-access-sc4sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:06.453588 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.444606 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-dshm" pod "1983d9b4-c329-4817-b2ff-960d8f42c6f5" (UID: "1983d9b4-c329-4817-b2ff-960d8f42c6f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:06.453588 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.453324 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tls-certs" pod "1983d9b4-c329-4817-b2ff-960d8f42c6f5" (UID: "1983d9b4-c329-4817-b2ff-960d8f42c6f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:06.464030 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.463989 5838 generic.go:356] "Generic (PLEG): container finished" podID="1983d9b4-c329-4817-b2ff-960d8f42c6f5" containerID="e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004" exitCode=0 Oct 30 16:51:06.464205 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.464145 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" event={"ID":"1983d9b4-c329-4817-b2ff-960d8f42c6f5","Type":"ContainerDied","Data":"e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004"} Oct 30 16:51:06.464205 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.464182 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" event={"ID":"1983d9b4-c329-4817-b2ff-960d8f42c6f5","Type":"ContainerDied","Data":"b89add359c555899756195d8af538ade4df246511a0c779b72203eac881f3832"} Oct 30 16:51:06.464205 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.464198 5838 scope.go:122] "RemoveContainer" containerID="e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004" Oct 30 16:51:06.464395 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.464323 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2" Oct 30 16:51:06.481609 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.479448 5838 scope.go:122] "RemoveContainer" containerID="e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004" Oct 30 16:51:06.481609 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:51:06.479771 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004\": container with ID starting with e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004 not found: ID does not exist" containerID="e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004" Oct 30 16:51:06.481609 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.479809 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004"} err="failed to get container status \"e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004\": rpc error: code = NotFound desc = could not find container \"e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004\": container with ID starting with e7cb97bdd9ea3182d0518ae7238b6e396fd2e9794bf4a89dc6e9949905f28004 not found: ID does not exist" Oct 30 16:51:06.515651 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.515589 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2"] Oct 30 16:51:06.518880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.518829 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0ab40f41-kserve-7574c557bf-8w9c2"] Oct 30 16:51:06.528768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.528580 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc4sz\" (UniqueName: \"kubernetes.io/projected/1983d9b4-c329-4817-b2ff-960d8f42c6f5-kube-api-access-sc4sz\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:06.528768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.528626 5838 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-home\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:06.528768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.528643 5838 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tls-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:06.528768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.528656 5838 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-dshm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:06.528768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.528668 5838 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-tmp-dir\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:06.528768 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:06.528685 5838 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1983d9b4-c329-4817-b2ff-960d8f42c6f5-model-cache\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:08.394547 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:08.394513 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1983d9b4-c329-4817-b2ff-960d8f42c6f5" path="/var/lib/kubelet/pods/1983d9b4-c329-4817-b2ff-960d8f42c6f5/volumes" Oct 30 16:51:13.010169 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:13.010134 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:51:18.168141 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.168100 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc"] Oct 30 16:51:18.168670 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.168631 5838 cpu_manager.go:414] "containerMap: removing container" podUID="a19bc095-b22a-44b3-8aa4-e6d4f988f946" containerName="main" Oct 30 16:51:18.168670 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.168647 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19bc095-b22a-44b3-8aa4-e6d4f988f946" containerName="main" Oct 30 16:51:18.168670 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.168665 5838 cpu_manager.go:414] "containerMap: removing container" podUID="1983d9b4-c329-4817-b2ff-960d8f42c6f5" containerName="main" Oct 30 16:51:18.168670 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.168671 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1983d9b4-c329-4817-b2ff-960d8f42c6f5" containerName="main" Oct 30 16:51:18.168867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.168693 5838 cpu_manager.go:414] "containerMap: removing container" podUID="0487b0d7-2a8b-4267-862c-e85749810588" containerName="revoke-keys" Oct 30 16:51:18.168867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.168699 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0487b0d7-2a8b-4267-862c-e85749810588" containerName="revoke-keys" Oct 30 16:51:18.168867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.168765 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="0487b0d7-2a8b-4267-862c-e85749810588" containerName="revoke-keys" Oct 30 16:51:18.168867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.168776 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19bc095-b22a-44b3-8aa4-e6d4f988f946" containerName="main" Oct 30 16:51:18.168867 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.168782 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="1983d9b4-c329-4817-b2ff-960d8f42c6f5" containerName="main" Oct 30 16:51:18.173259 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.173235 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:18.191248 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.191218 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc"] Oct 30 16:51:18.200590 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.200533 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:51:18.200590 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.200535 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:51:18.200590 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.200604 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:51:18.200590 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.200659 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rmkzn\"" Oct 30 16:51:18.266129 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.266091 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4q5s\" (UniqueName: \"kubernetes.io/projected/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-kube-api-access-l4q5s\") pod \"maas-api-revoke-sub-84e476db-t5qxc\" (UID: \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:18.266299 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.266137 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84e476db-t5qxc\" (UID: \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:18.266299 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.266227 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84e476db-t5qxc\" (UID: \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:18.367514 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.367473 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4q5s\" (UniqueName: \"kubernetes.io/projected/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-kube-api-access-l4q5s\") pod \"maas-api-revoke-sub-84e476db-t5qxc\" (UID: \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:18.367750 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.367531 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84e476db-t5qxc\" (UID: \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:18.367750 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.367604 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84e476db-t5qxc\" (UID: \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:18.368317 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.368291 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84e476db-t5qxc\" (UID: \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:18.370184 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.370161 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84e476db-t5qxc\" (UID: \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:18.390627 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.390601 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4q5s\" (UniqueName: \"kubernetes.io/projected/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-kube-api-access-l4q5s\") pod \"maas-api-revoke-sub-84e476db-t5qxc\" (UID: \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:18.487343 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.487311 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:18.787533 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:18.787455 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc"] Oct 30 16:51:18.790843 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:51:18.790807 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44121bf0_c5b3_44f4_b7a6_d0819a7a4133.slice/crio-e9dffe5b4b57cdbd1467241ff6abe408983ab45d067316333169af17d4686a32 WatchSource:0}: Error finding container e9dffe5b4b57cdbd1467241ff6abe408983ab45d067316333169af17d4686a32: Status 404 returned error can't find the container with id e9dffe5b4b57cdbd1467241ff6abe408983ab45d067316333169af17d4686a32 Oct 30 16:51:19.560271 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:19.560240 5838 generic.go:356] "Generic (PLEG): container finished" podID="44121bf0-c5b3-44f4-b7a6-d0819a7a4133" containerID="1cc4ebb185a556ce3c9c352b7f9d2f38e6a10a7f4ded145901290b05e78919b9" exitCode=0 Oct 30 16:51:19.560609 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:19.560322 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" event={"ID":"44121bf0-c5b3-44f4-b7a6-d0819a7a4133","Type":"ContainerDied","Data":"1cc4ebb185a556ce3c9c352b7f9d2f38e6a10a7f4ded145901290b05e78919b9"} Oct 30 16:51:19.560609 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:19.560400 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" event={"ID":"44121bf0-c5b3-44f4-b7a6-d0819a7a4133","Type":"ContainerStarted","Data":"e9dffe5b4b57cdbd1467241ff6abe408983ab45d067316333169af17d4686a32"} Oct 30 16:51:20.702613 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:20.702586 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:20.792341 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:20.792308 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-openshift-ca\") pod \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\" (UID: \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\") " Oct 30 16:51:20.792523 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:20.792351 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-kube-api-access-l4q5s\" (UniqueName: \"kubernetes.io/projected/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-kube-api-access-l4q5s\") pod \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\" (UID: \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\") " Oct 30 16:51:20.792523 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:20.792439 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-xks-ca\") pod \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\" (UID: \"44121bf0-c5b3-44f4-b7a6-d0819a7a4133\") " Oct 30 16:51:20.792773 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:20.792747 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-openshift-ca" pod "44121bf0-c5b3-44f4-b7a6-d0819a7a4133" (UID: "44121bf0-c5b3-44f4-b7a6-d0819a7a4133"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:20.794543 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:20.794503 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-kube-api-access-l4q5s" pod "44121bf0-c5b3-44f4-b7a6-d0819a7a4133" (UID: "44121bf0-c5b3-44f4-b7a6-d0819a7a4133"). InnerVolumeSpecName "kube-api-access-l4q5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:20.794687 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:20.794598 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-xks-ca" pod "44121bf0-c5b3-44f4-b7a6-d0819a7a4133" (UID: "44121bf0-c5b3-44f4-b7a6-d0819a7a4133"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:20.893386 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:20.893291 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-xks-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:20.893386 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:20.893327 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-maas-api-openshift-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:20.893386 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:20.893341 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4q5s\" (UniqueName: \"kubernetes.io/projected/44121bf0-c5b3-44f4-b7a6-d0819a7a4133-kube-api-access-l4q5s\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:21.571809 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:21.571774 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" Oct 30 16:51:21.572013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:21.571805 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84e476db-t5qxc" event={"ID":"44121bf0-c5b3-44f4-b7a6-d0819a7a4133","Type":"ContainerDied","Data":"e9dffe5b4b57cdbd1467241ff6abe408983ab45d067316333169af17d4686a32"} Oct 30 16:51:21.572013 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:21.571860 5838 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9dffe5b4b57cdbd1467241ff6abe408983ab45d067316333169af17d4686a32" Oct 30 16:51:24.034389 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:24.034333 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx"] Oct 30 16:51:24.045248 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:24.045207 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-thntx"] Oct 30 16:51:24.394600 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:24.394523 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d8cca2-4bc2-4f10-8329-5a809efed30e" path="/var/lib/kubelet/pods/97d8cca2-4bc2-4f10-8329-5a809efed30e/volumes" Oct 30 16:51:29.408408 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:29.408372 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:51:30.598902 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.598867 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5"] Oct 30 16:51:30.599314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.599287 5838 cpu_manager.go:414] "containerMap: removing container" podUID="44121bf0-c5b3-44f4-b7a6-d0819a7a4133" containerName="revoke-keys" Oct 30 16:51:30.599314 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.599300 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44121bf0-c5b3-44f4-b7a6-d0819a7a4133" containerName="revoke-keys" Oct 30 16:51:30.599395 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.599385 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="44121bf0-c5b3-44f4-b7a6-d0819a7a4133" containerName="revoke-keys" Oct 30 16:51:30.603891 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.603868 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.619626 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.619599 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-xbwnt"] Oct 30 16:51:30.623360 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.623340 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.640009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.634169 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dqwfr\"" Oct 30 16:51:30.640009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.634187 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 16:51:30.640009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.634169 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 16:51:30.640009 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.637404 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5"] Oct 30 16:51:30.643763 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.643688 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-xbwnt"] Oct 30 16:51:30.687371 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.687340 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/035aeb7c-9167-4780-953d-6a5ab9d2be55-metrics-certs\") pod \"maas-api-595cc648c-xbwnt\" (UID: \"035aeb7c-9167-4780-953d-6a5ab9d2be55\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.687555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.687398 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-maas-api-tls\") pod \"maas-api-5c48877c75-gmvx5\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.687555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.687469 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-openshift-service-ca\") pod \"maas-api-5c48877c75-gmvx5\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.687555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.687502 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-metrics-certs\") pod \"maas-api-5c48877c75-gmvx5\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.687555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.687531 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w89w\" (UniqueName: \"kubernetes.io/projected/035aeb7c-9167-4780-953d-6a5ab9d2be55-kube-api-access-6w89w\") pod \"maas-api-595cc648c-xbwnt\" (UID: \"035aeb7c-9167-4780-953d-6a5ab9d2be55\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.687995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.687617 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gm88\" (UniqueName: \"kubernetes.io/projected/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-kube-api-access-6gm88\") pod \"maas-api-5c48877c75-gmvx5\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.687995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.687641 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035aeb7c-9167-4780-953d-6a5ab9d2be55-maas-api-tls\") pod \"maas-api-595cc648c-xbwnt\" (UID: \"035aeb7c-9167-4780-953d-6a5ab9d2be55\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.687995 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.687682 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035aeb7c-9167-4780-953d-6a5ab9d2be55-openshift-service-ca\") pod \"maas-api-595cc648c-xbwnt\" (UID: \"035aeb7c-9167-4780-953d-6a5ab9d2be55\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.697062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.697018 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:51:30.697062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.697022 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:51:30.788365 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.788331 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-maas-api-tls\") pod \"maas-api-5c48877c75-gmvx5\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.788562 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.788371 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-openshift-service-ca\") pod \"maas-api-5c48877c75-gmvx5\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.788562 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.788393 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-metrics-certs\") pod \"maas-api-5c48877c75-gmvx5\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.788562 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.788416 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w89w\" (UniqueName: \"kubernetes.io/projected/035aeb7c-9167-4780-953d-6a5ab9d2be55-kube-api-access-6w89w\") pod \"maas-api-595cc648c-xbwnt\" (UID: \"035aeb7c-9167-4780-953d-6a5ab9d2be55\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.788562 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.788461 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gm88\" (UniqueName: \"kubernetes.io/projected/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-kube-api-access-6gm88\") pod \"maas-api-5c48877c75-gmvx5\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.788562 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.788478 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035aeb7c-9167-4780-953d-6a5ab9d2be55-maas-api-tls\") pod \"maas-api-595cc648c-xbwnt\" (UID: \"035aeb7c-9167-4780-953d-6a5ab9d2be55\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.788562 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.788492 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035aeb7c-9167-4780-953d-6a5ab9d2be55-openshift-service-ca\") pod \"maas-api-595cc648c-xbwnt\" (UID: \"035aeb7c-9167-4780-953d-6a5ab9d2be55\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.788562 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.788547 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/035aeb7c-9167-4780-953d-6a5ab9d2be55-metrics-certs\") pod \"maas-api-595cc648c-xbwnt\" (UID: \"035aeb7c-9167-4780-953d-6a5ab9d2be55\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.789292 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.789263 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-openshift-service-ca\") pod \"maas-api-5c48877c75-gmvx5\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.789555 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.789530 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/035aeb7c-9167-4780-953d-6a5ab9d2be55-openshift-service-ca\") pod \"maas-api-595cc648c-xbwnt\" (UID: \"035aeb7c-9167-4780-953d-6a5ab9d2be55\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.791126 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.791101 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-maas-api-tls\") pod \"maas-api-5c48877c75-gmvx5\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.791231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.791161 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-metrics-certs\") pod \"maas-api-5c48877c75-gmvx5\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.791231 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.791182 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/035aeb7c-9167-4780-953d-6a5ab9d2be55-metrics-certs\") pod \"maas-api-595cc648c-xbwnt\" (UID: \"035aeb7c-9167-4780-953d-6a5ab9d2be55\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.791313 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.791235 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/035aeb7c-9167-4780-953d-6a5ab9d2be55-maas-api-tls\") pod \"maas-api-595cc648c-xbwnt\" (UID: \"035aeb7c-9167-4780-953d-6a5ab9d2be55\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.811335 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.811311 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w89w\" (UniqueName: \"kubernetes.io/projected/035aeb7c-9167-4780-953d-6a5ab9d2be55-kube-api-access-6w89w\") pod \"maas-api-595cc648c-xbwnt\" (UID: \"035aeb7c-9167-4780-953d-6a5ab9d2be55\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:30.811519 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.811498 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gm88\" (UniqueName: \"kubernetes.io/projected/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-kube-api-access-6gm88\") pod \"maas-api-5c48877c75-gmvx5\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.917962 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.917864 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:30.940273 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:30.940243 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:31.221401 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:31.221194 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5"] Oct 30 16:51:31.224275 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:51:31.224247 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda254a4b1_b22b_4712_bdb6_a369fa6fd8ee.slice/crio-3ad32dbf13c14f9e1dc1648a6fa3d6cec064872d807f874b41c24a47f8c1cdeb WatchSource:0}: Error finding container 3ad32dbf13c14f9e1dc1648a6fa3d6cec064872d807f874b41c24a47f8c1cdeb: Status 404 returned error can't find the container with id 3ad32dbf13c14f9e1dc1648a6fa3d6cec064872d807f874b41c24a47f8c1cdeb Oct 30 16:51:31.225605 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:31.225587 5838 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:51:31.261528 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:31.261497 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-xbwnt"] Oct 30 16:51:31.629357 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:31.629323 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" event={"ID":"035aeb7c-9167-4780-953d-6a5ab9d2be55","Type":"ContainerStarted","Data":"7a201627d9233049a6b5537f5cebb3fb33ee9649c2a5eccd345cbefb57743c4b"} Oct 30 16:51:31.630680 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:31.630651 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" event={"ID":"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee","Type":"ContainerStarted","Data":"3ad32dbf13c14f9e1dc1648a6fa3d6cec064872d807f874b41c24a47f8c1cdeb"} Oct 30 16:51:31.799217 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:51:31.799171 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:51:32.637488 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:32.637442 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" event={"ID":"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee","Type":"ContainerStarted","Data":"f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903"} Oct 30 16:51:32.637905 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:32.637519 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:32.639117 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:32.639093 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" event={"ID":"035aeb7c-9167-4780-953d-6a5ab9d2be55","Type":"ContainerStarted","Data":"3c7958749054d4bd3436619dcfbc09cb1837ccba6a034d7133135a227c6d50fb"} Oct 30 16:51:32.639259 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:32.639242 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:32.680117 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:32.680071 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" podStartSLOduration=2.330187561 podStartE2EDuration="2.680058683s" totalImagesPullingTime="349.871122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:32.678059161 +0000 UTC m=+1854.426648583" watchObservedRunningTime="2026-10-30 16:51:32.680058683 +0000 UTC m=+1854.428648104" Oct 30 16:51:32.720419 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:32.720362 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" podStartSLOduration=2.2665842720000002 podStartE2EDuration="2.720348369s" totalImagesPullingTime="453.764097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:32.719127252 +0000 UTC m=+1854.467716660" watchObservedRunningTime="2026-10-30 16:51:32.720348369 +0000 UTC m=+1854.468937756" Oct 30 16:51:38.649986 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:38.649944 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:38.650445 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:38.650182 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-xbwnt" Oct 30 16:51:38.766089 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:38.766055 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5"] Oct 30 16:51:38.766309 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:38.766282 5838 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" podUID="a254a4b1-b22b-4712-bdb6-a369fa6fd8ee" containerName="maas-api" containerID="cri-o://f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903" gracePeriod=30 Oct 30 16:51:39.041307 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.041283 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:39.064767 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.064738 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-metrics-certs\") pod \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " Oct 30 16:51:39.064896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.064809 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-openshift-service-ca\") pod \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " Oct 30 16:51:39.064896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.064866 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-kube-api-access-6gm88\" (UniqueName: \"kubernetes.io/projected/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-kube-api-access-6gm88\") pod \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " Oct 30 16:51:39.064896 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.064888 5838 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-maas-api-tls\") pod \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\" (UID: \"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee\") " Oct 30 16:51:39.065216 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.065189 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-openshift-service-ca" pod "a254a4b1-b22b-4712-bdb6-a369fa6fd8ee" (UID: "a254a4b1-b22b-4712-bdb6-a369fa6fd8ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:39.067028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.066998 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-metrics-certs" pod "a254a4b1-b22b-4712-bdb6-a369fa6fd8ee" (UID: "a254a4b1-b22b-4712-bdb6-a369fa6fd8ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:39.067028 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.067023 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-kube-api-access-6gm88" pod "a254a4b1-b22b-4712-bdb6-a369fa6fd8ee" (UID: "a254a4b1-b22b-4712-bdb6-a369fa6fd8ee"). InnerVolumeSpecName "kube-api-access-6gm88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:39.067208 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.067092 5838 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-maas-api-tls" pod "a254a4b1-b22b-4712-bdb6-a369fa6fd8ee" (UID: "a254a4b1-b22b-4712-bdb6-a369fa6fd8ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:39.165800 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.165766 5838 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gm88\" (UniqueName: \"kubernetes.io/projected/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-kube-api-access-6gm88\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:39.165800 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.165795 5838 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-maas-api-tls\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:39.165800 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.165807 5838 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-metrics-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:39.166062 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.165820 5838 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee-openshift-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Oct 30 16:51:39.340942 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.340900 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:51:39.678366 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.678282 5838 generic.go:356] "Generic (PLEG): container finished" podID="a254a4b1-b22b-4712-bdb6-a369fa6fd8ee" containerID="f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903" exitCode=0 Oct 30 16:51:39.678366 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.678325 5838 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" Oct 30 16:51:39.678796 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.678363 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" event={"ID":"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee","Type":"ContainerDied","Data":"f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903"} Oct 30 16:51:39.678796 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.678419 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5" event={"ID":"a254a4b1-b22b-4712-bdb6-a369fa6fd8ee","Type":"ContainerDied","Data":"3ad32dbf13c14f9e1dc1648a6fa3d6cec064872d807f874b41c24a47f8c1cdeb"} Oct 30 16:51:39.678796 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.678448 5838 scope.go:122] "RemoveContainer" containerID="f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903" Oct 30 16:51:39.690259 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.690236 5838 scope.go:122] "RemoveContainer" containerID="f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903" Oct 30 16:51:39.690643 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:51:39.690622 5838 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903\": container with ID starting with f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903 not found: ID does not exist" containerID="f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903" Oct 30 16:51:39.690705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.690654 5838 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903"} err="failed to get container status \"f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903\": rpc error: code = NotFound desc = could not find container \"f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903\": container with ID starting with f6b3b0594f5862c20cb63ceea6f36c273f95442469495985b5a575e0e4f6a903 not found: ID does not exist" Oct 30 16:51:39.732392 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.732348 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5"] Oct 30 16:51:39.742917 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:39.742877 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-gmvx5"] Oct 30 16:51:40.395736 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:40.395702 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a254a4b1-b22b-4712-bdb6-a369fa6fd8ee" path="/var/lib/kubelet/pods/a254a4b1-b22b-4712-bdb6-a369fa6fd8ee/volumes" Oct 30 16:51:41.021994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:41.019527 5838 scope.go:122] "RemoveContainer" containerID="8a97d7a9d5e0645112fec15916d9a3b90a3d3edbe1ba7d570ce41dd525c58586" Oct 30 16:51:47.449531 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:47.449490 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:51:50.178997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:50.178097 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:51:50.178997 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:50.178187 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-x6sns" podUID="fecda1cb-458a-4c3e-b48f-a0e6d82c1995" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:51:50.248996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:50.246087 5838 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:51:50.248996 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:51:50.246159 5838 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9" podUID="006df6fa-0ef9-4007-869e-0a251d92c372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:00.838960 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:00.838925 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:52:01.797209 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:52:01.797170 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:52:31.796429 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:52:31.796348 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:52:37.933994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:37.933930 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zjffw"] Oct 30 16:52:41.115880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:41.115846 5838 scope.go:122] "RemoveContainer" containerID="ae502831374f96a840892c51cca3192731278818ea7204da70f7e226bf87cc83" Oct 30 16:52:49.239917 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:49.239886 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67b6f84467-248zx_4943e10f-4269-4eba-a25d-4996fa2fb20a/manager/0.log" Oct 30 16:52:50.230769 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:50.230734 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85bb7dd889-2mmgt_fb90bed1-543d-499e-a996-85e01f7d3bcb/manager/0.log" Oct 30 16:52:51.383541 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:51.383510 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dcb75f99-swmnc_c64a302c-08f6-4b2b-bd1b-c6bb63a51e27/copy-manifests/0.log" Oct 30 16:52:51.418437 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:51.418408 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dcb75f99-swmnc_c64a302c-08f6-4b2b-bd1b-c6bb63a51e27/manager/0.log" Oct 30 16:52:52.907266 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:52.907237 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-xbwnt_035aeb7c-9167-4780-953d-6a5ab9d2be55/maas-api/0.log" Oct 30 16:52:53.315092 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:53.315062 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-skhd4_0487b0d7-2a8b-4267-862c-e85749810588/revoke-keys/0.log" Oct 30 16:52:53.513102 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:53.513074 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-4djbs_6342b4fe-4d83-4dc6-94ac-25cb432ae497/revoke-keys/0.log" Oct 30 16:52:53.899775 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:53.899735 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-84e476db-t5qxc_44121bf0-c5b3-44f4-b7a6-d0819a7a4133/revoke-keys/0.log" Oct 30 16:52:54.268081 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:54.268048 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-bjvdn_18bd0d59-647e-4683-9019-82d6a95ade7b/revoke-keys/0.log" Oct 30 16:52:54.462994 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:54.462942 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-8q2b4_e78dd4f4-4b65-4a24-9cc1-497f54756e2d/revoke-keys/0.log" Oct 30 16:52:56.697385 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:56.697358 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-srmxg_e32488e8-fdae-4b20-906b-2bb8d48be984/manager/0.log" Oct 30 16:52:56.900692 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:56.900655 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-x5dpg_3bcaa3b3-de70-4a7c-82f6-5ec295abcfee/manager/0.log" Oct 30 16:52:57.568905 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:57.568874 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zjffw_89846f13-eb01-4f1e-9abb-2d5ad16c2265/limitador/0.log" Oct 30 16:52:57.778880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:57.778844 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-rpdmp_f0959ec1-57e5-4472-875b-7ebfec809d5d/manager/0.log" Oct 30 16:52:58.639697 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:58.639664 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-vdpqj_5c2f91bc-56b8-45bb-9b39-876213305120/istio-proxy/0.log" Oct 30 16:52:58.825901 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:58.825867 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vl9xp_bb333f2b-2469-4dc5-bb87-2e633640b18b/discovery/0.log" Oct 30 16:52:59.045410 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:52:59.045380 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dcc5c66cd-7dvjr_035709ff-bb86-42f9-952b-95e17119d171/kube-auth-proxy/0.log" Oct 30 16:53:00.077541 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:00.077509 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-djmdh_3465e7a5-d6cc-4c8c-9052-29034ba0847f/istio-proxy/1.log" Oct 30 16:53:00.294657 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:00.294616 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9_006df6fa-0ef9-4007-869e-0a251d92c372/istio-proxy/0.log" Oct 30 16:53:01.091839 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:01.091806 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-x6sns_fecda1cb-458a-4c3e-b48f-a0e6d82c1995/istio-proxy/0.log" Oct 30 16:53:01.504936 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:01.504908 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-sv5ln_8aeeae52-0ed0-4b70-9da0-640a6dfac794/payload-pre-processing/0.log" Oct 30 16:53:01.697729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:01.697701 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9hs5j_72f776e7-ae4d-4ccc-ab01-d107e80df07e/payload-processing/0.log" Oct 30 16:53:01.796221 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:53:01.796127 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:53:02.408846 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:02.408815 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-djmdh_3465e7a5-d6cc-4c8c-9052-29034ba0847f/istio-proxy/0.log" Oct 30 16:53:03.956860 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:03.956825 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn_5a60bebc-f990-43e5-b655-458e6bf057c3/main/0.log" Oct 30 16:53:04.176016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:04.175982 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq_36e0fb5d-ff56-4fd3-a4a6-683306972ef7/main/0.log" Oct 30 16:53:14.028025 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:14.027985 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4"] Oct 30 16:53:14.039320 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:14.039286 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8q2b4"] Oct 30 16:53:14.394962 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:14.394879 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78dd4f4-4b65-4a24-9cc1-497f54756e2d" path="/var/lib/kubelet/pods/e78dd4f4-4b65-4a24-9cc1-497f54756e2d/volumes" Oct 30 16:53:14.765813 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:14.765772 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67b6f84467-248zx_4943e10f-4269-4eba-a25d-4996fa2fb20a/manager/0.log" Oct 30 16:53:15.646243 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:15.646208 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85bb7dd889-2mmgt_fb90bed1-543d-499e-a996-85e01f7d3bcb/manager/0.log" Oct 30 16:53:16.786105 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:16.786076 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dcb75f99-swmnc_c64a302c-08f6-4b2b-bd1b-c6bb63a51e27/copy-manifests/0.log" Oct 30 16:53:16.820546 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:16.820510 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dcb75f99-swmnc_c64a302c-08f6-4b2b-bd1b-c6bb63a51e27/manager/0.log" Oct 30 16:53:18.047444 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:18.047411 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-xbwnt_035aeb7c-9167-4780-953d-6a5ab9d2be55/maas-api/0.log" Oct 30 16:53:18.445707 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:18.445626 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-skhd4_0487b0d7-2a8b-4267-862c-e85749810588/revoke-keys/0.log" Oct 30 16:53:18.631726 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:18.631696 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-4djbs_6342b4fe-4d83-4dc6-94ac-25cb432ae497/revoke-keys/0.log" Oct 30 16:53:19.018409 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:19.018378 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-84e476db-t5qxc_44121bf0-c5b3-44f4-b7a6-d0819a7a4133/revoke-keys/0.log" Oct 30 16:53:19.405719 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:19.405631 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-bjvdn_18bd0d59-647e-4683-9019-82d6a95ade7b/revoke-keys/0.log" Oct 30 16:53:21.227780 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:21.227738 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-srmxg_e32488e8-fdae-4b20-906b-2bb8d48be984/manager/0.log" Oct 30 16:53:21.437179 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:21.437153 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-x5dpg_3bcaa3b3-de70-4a7c-82f6-5ec295abcfee/manager/0.log" Oct 30 16:53:22.115174 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:22.115144 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zjffw_89846f13-eb01-4f1e-9abb-2d5ad16c2265/limitador/0.log" Oct 30 16:53:22.319705 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:22.319677 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-rpdmp_f0959ec1-57e5-4472-875b-7ebfec809d5d/manager/0.log" Oct 30 16:53:22.928924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:22.928893 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-vdpqj_5c2f91bc-56b8-45bb-9b39-876213305120/istio-proxy/0.log" Oct 30 16:53:23.128927 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:23.128893 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vl9xp_bb333f2b-2469-4dc5-bb87-2e633640b18b/discovery/0.log" Oct 30 16:53:23.368601 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:23.368566 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dcc5c66cd-7dvjr_035709ff-bb86-42f9-952b-95e17119d171/kube-auth-proxy/0.log" Oct 30 16:53:24.417898 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:24.413353 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-djmdh_3465e7a5-d6cc-4c8c-9052-29034ba0847f/istio-proxy/1.log" Oct 30 16:53:24.632267 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:24.632232 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9_006df6fa-0ef9-4007-869e-0a251d92c372/istio-proxy/0.log" Oct 30 16:53:25.437104 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:25.437072 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-x6sns_fecda1cb-458a-4c3e-b48f-a0e6d82c1995/istio-proxy/0.log" Oct 30 16:53:25.875802 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:25.875770 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-sv5ln_8aeeae52-0ed0-4b70-9da0-640a6dfac794/payload-pre-processing/0.log" Oct 30 16:53:26.085736 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:26.085705 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9hs5j_72f776e7-ae4d-4ccc-ab01-d107e80df07e/payload-processing/0.log" Oct 30 16:53:27.873546 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:27.873512 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-dhfxn_5a60bebc-f990-43e5-b655-458e6bf057c3/main/0.log" Oct 30 16:53:28.079194 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:28.079164 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-zs5lq_36e0fb5d-ff56-4fd3-a4a6-683306972ef7/main/0.log" Oct 30 16:53:31.034037 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:31.034001 5838 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn"] Oct 30 16:53:31.044880 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:31.044851 5838 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-bjvdn"] Oct 30 16:53:31.796555 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:53:31.796514 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:53:32.394091 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:53:32.394053 5838 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18bd0d59-647e-4683-9019-82d6a95ade7b" path="/var/lib/kubelet/pods/18bd0d59-647e-4683-9019-82d6a95ade7b/volumes" Oct 30 16:54:01.797196 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:54:01.797153 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:54:07.715628 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:07.715592 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85bb7dd889-2mmgt_fb90bed1-543d-499e-a996-85e01f7d3bcb/manager/0.log" Oct 30 16:54:08.025690 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:08.025652 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85bb7dd889-2mmgt_fb90bed1-543d-499e-a996-85e01f7d3bcb/manager/0.log" Oct 30 16:54:08.823265 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:08.823235 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9hs5j_72f776e7-ae4d-4ccc-ab01-d107e80df07e/payload-processing/0.log" Oct 30 16:54:09.021763 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:09.021736 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9hs5j_72f776e7-ae4d-4ccc-ab01-d107e80df07e/payload-processing/0.log" Oct 30 16:54:31.797565 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:54:31.797528 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:54:41.149215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:41.149179 5838 scope.go:122] "RemoveContainer" containerID="9d3d1e2d420e6d1fa4d7ea1706a3e236eaedb6c7d60944259f16f355616d771c" Oct 30 16:54:41.158384 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:41.158335 5838 scope.go:122] "RemoveContainer" containerID="d5a4267283b5cd55a122b379ee305e16ff59ab64576e8309d4524adcc3cc6de5" Oct 30 16:54:57.473130 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:57.473090 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85bb7dd889-2mmgt_fb90bed1-543d-499e-a996-85e01f7d3bcb/manager/0.log" Oct 30 16:54:57.759435 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:57.759400 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85bb7dd889-2mmgt_fb90bed1-543d-499e-a996-85e01f7d3bcb/manager/0.log" Oct 30 16:54:58.529873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:58.529832 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9hs5j_72f776e7-ae4d-4ccc-ab01-d107e80df07e/payload-processing/0.log" Oct 30 16:54:58.724956 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:58.724922 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9hs5j_72f776e7-ae4d-4ccc-ab01-d107e80df07e/payload-processing/0.log" Oct 30 16:54:59.055631 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.055596 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dcxv9/must-gather-bf8gq"] Oct 30 16:54:59.056106 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.056090 5838 cpu_manager.go:414] "containerMap: removing container" podUID="a254a4b1-b22b-4712-bdb6-a369fa6fd8ee" containerName="maas-api" Oct 30 16:54:59.056163 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.056108 5838 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a254a4b1-b22b-4712-bdb6-a369fa6fd8ee" containerName="maas-api" Oct 30 16:54:59.056200 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.056176 5838 memory_manager.go:365] "RemoveStaleState removing state" podUID="a254a4b1-b22b-4712-bdb6-a369fa6fd8ee" containerName="maas-api" Oct 30 16:54:59.060642 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.060624 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dcxv9/must-gather-bf8gq" Oct 30 16:54:59.074826 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.074801 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dcxv9\"/\"kube-root-ca.crt\"" Oct 30 16:54:59.086016 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.085988 5838 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dcxv9\"/\"openshift-service-ca.crt\"" Oct 30 16:54:59.086170 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.086095 5838 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dcxv9\"/\"default-dockercfg-sj5n4\"" Oct 30 16:54:59.097798 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.097758 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dcxv9/must-gather-bf8gq"] Oct 30 16:54:59.127060 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.127016 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/857a3597-e859-486e-9f67-3dc55eccf0fa-must-gather-output\") pod \"must-gather-bf8gq\" (UID: \"857a3597-e859-486e-9f67-3dc55eccf0fa\") " pod="openshift-must-gather-dcxv9/must-gather-bf8gq" Oct 30 16:54:59.127060 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.127055 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9t2k\" (UniqueName: \"kubernetes.io/projected/857a3597-e859-486e-9f67-3dc55eccf0fa-kube-api-access-s9t2k\") pod \"must-gather-bf8gq\" (UID: \"857a3597-e859-486e-9f67-3dc55eccf0fa\") " pod="openshift-must-gather-dcxv9/must-gather-bf8gq" Oct 30 16:54:59.228685 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.228640 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/857a3597-e859-486e-9f67-3dc55eccf0fa-must-gather-output\") pod \"must-gather-bf8gq\" (UID: \"857a3597-e859-486e-9f67-3dc55eccf0fa\") " pod="openshift-must-gather-dcxv9/must-gather-bf8gq" Oct 30 16:54:59.228685 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.228682 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9t2k\" (UniqueName: \"kubernetes.io/projected/857a3597-e859-486e-9f67-3dc55eccf0fa-kube-api-access-s9t2k\") pod \"must-gather-bf8gq\" (UID: \"857a3597-e859-486e-9f67-3dc55eccf0fa\") " pod="openshift-must-gather-dcxv9/must-gather-bf8gq" Oct 30 16:54:59.229135 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.229115 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/857a3597-e859-486e-9f67-3dc55eccf0fa-must-gather-output\") pod \"must-gather-bf8gq\" (UID: \"857a3597-e859-486e-9f67-3dc55eccf0fa\") " pod="openshift-must-gather-dcxv9/must-gather-bf8gq" Oct 30 16:54:59.252285 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.252256 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9t2k\" (UniqueName: \"kubernetes.io/projected/857a3597-e859-486e-9f67-3dc55eccf0fa-kube-api-access-s9t2k\") pod \"must-gather-bf8gq\" (UID: \"857a3597-e859-486e-9f67-3dc55eccf0fa\") " pod="openshift-must-gather-dcxv9/must-gather-bf8gq" Oct 30 16:54:59.375215 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.375128 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dcxv9/must-gather-bf8gq" Oct 30 16:54:59.679910 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.679832 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dcxv9/must-gather-bf8gq"] Oct 30 16:54:59.683709 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:54:59.683669 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod857a3597_e859_486e_9f67_3dc55eccf0fa.slice/crio-f7427d2eac39f41484d02c3f7d9ba8f52dbd26ddcef30917e2b049e3ca311a30 WatchSource:0}: Error finding container f7427d2eac39f41484d02c3f7d9ba8f52dbd26ddcef30917e2b049e3ca311a30: Status 404 returned error can't find the container with id f7427d2eac39f41484d02c3f7d9ba8f52dbd26ddcef30917e2b049e3ca311a30 Oct 30 16:54:59.851526 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:54:59.851484 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcxv9/must-gather-bf8gq" event={"ID":"857a3597-e859-486e-9f67-3dc55eccf0fa","Type":"ContainerStarted","Data":"f7427d2eac39f41484d02c3f7d9ba8f52dbd26ddcef30917e2b049e3ca311a30"} Oct 30 16:55:00.860389 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:00.860348 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcxv9/must-gather-bf8gq" event={"ID":"857a3597-e859-486e-9f67-3dc55eccf0fa","Type":"ContainerStarted","Data":"1da73d8e7922499ce04be4734a0126b3659252892a3494189b82b71c6d96106a"} Oct 30 16:55:00.860893 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:00.860401 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcxv9/must-gather-bf8gq" event={"ID":"857a3597-e859-486e-9f67-3dc55eccf0fa","Type":"ContainerStarted","Data":"bfc3ff5232d5e37c4c263a04ecbb5a5c6472c10a1df8ecc0d2900fbf3baf95cc"} Oct 30 16:55:00.901188 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:00.901126 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dcxv9/must-gather-bf8gq" podStartSLOduration=1.11992953 podStartE2EDuration="1.901110108s" totalImagesPullingTime="781.180578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:00.899806536 +0000 UTC m=+2062.648395924" watchObservedRunningTime="2026-10-30 16:55:00.901110108 +0000 UTC m=+2062.649699536" Oct 30 16:55:01.799489 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:55:01.799450 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor" Oct 30 16:55:03.686471 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:03.685157 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lt2pz_b36b8f12-981e-46a8-b422-632babc5d4a7/global-pull-secret-syncer/0.log" Oct 30 16:55:04.698999 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:04.696925 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h8rzc_30634e3d-5f01-4b9b-88ad-03893a2b444e/konnectivity-agent/0.log" Oct 30 16:55:04.962048 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:04.961223 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-219.ec2.internal_a8e5ad7f7f3345c1093a31c3b5fc9d86/haproxy/0.log" Oct 30 16:55:10.825385 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:10.825356 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-srmxg_e32488e8-fdae-4b20-906b-2bb8d48be984/manager/0.log" Oct 30 16:55:10.950064 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:10.949938 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-x5dpg_3bcaa3b3-de70-4a7c-82f6-5ec295abcfee/manager/0.log" Oct 30 16:55:11.318988 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:11.317742 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zjffw_89846f13-eb01-4f1e-9abb-2d5ad16c2265/limitador/0.log" Oct 30 16:55:11.477106 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:11.477041 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-rpdmp_f0959ec1-57e5-4472-875b-7ebfec809d5d/manager/0.log" Oct 30 16:55:13.439845 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.439809 5838 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s"] Oct 30 16:55:13.446047 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.446011 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.468128 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.468095 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s"] Oct 30 16:55:13.595620 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.595585 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bsmp\" (UniqueName: \"kubernetes.io/projected/4a5872c4-c7aa-4c85-97fc-4b956ad824de-kube-api-access-7bsmp\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.595800 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.595643 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a5872c4-c7aa-4c85-97fc-4b956ad824de-podres\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.595800 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.595713 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a5872c4-c7aa-4c85-97fc-4b956ad824de-sys\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.595800 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.595772 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a5872c4-c7aa-4c85-97fc-4b956ad824de-proc\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.595931 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.595839 5838 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a5872c4-c7aa-4c85-97fc-4b956ad824de-lib-modules\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.696743 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.696649 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a5872c4-c7aa-4c85-97fc-4b956ad824de-podres\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.697234 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.697131 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a5872c4-c7aa-4c85-97fc-4b956ad824de-sys\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.697399 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.697381 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a5872c4-c7aa-4c85-97fc-4b956ad824de-proc\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.697806 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.697690 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a5872c4-c7aa-4c85-97fc-4b956ad824de-lib-modules\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.697998 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.697053 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a5872c4-c7aa-4c85-97fc-4b956ad824de-podres\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.698130 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.697546 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a5872c4-c7aa-4c85-97fc-4b956ad824de-proc\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.698628 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.697202 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a5872c4-c7aa-4c85-97fc-4b956ad824de-sys\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.698757 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.697771 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a5872c4-c7aa-4c85-97fc-4b956ad824de-lib-modules\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.698897 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.698879 5838 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bsmp\" (UniqueName: \"kubernetes.io/projected/4a5872c4-c7aa-4c85-97fc-4b956ad824de-kube-api-access-7bsmp\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.725253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.721868 5838 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bsmp\" (UniqueName: \"kubernetes.io/projected/4a5872c4-c7aa-4c85-97fc-4b956ad824de-kube-api-access-7bsmp\") pod \"perf-node-gather-daemonset-l765s\" (UID: \"4a5872c4-c7aa-4c85-97fc-4b956ad824de\") " pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.771147 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.771105 5838 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:13.978876 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:13.978788 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5l2t5_be9a1b81-45d0-46a0-a771-84d88046f28e/kube-state-metrics/0.log" Oct 30 16:55:14.081992 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:14.081350 5838 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s"] Oct 30 16:55:14.096334 ip-10-0-137-219 kubenswrapper[5838]: W1030 16:55:14.096285 5838 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4a5872c4_c7aa_4c85_97fc_4b956ad824de.slice/crio-af0161ca8e0a0b6c9d6af389b77db10b30aaefa9c294d10d8e8abf43b3aede36 WatchSource:0}: Error finding container af0161ca8e0a0b6c9d6af389b77db10b30aaefa9c294d10d8e8abf43b3aede36: Status 404 returned error can't find the container with id af0161ca8e0a0b6c9d6af389b77db10b30aaefa9c294d10d8e8abf43b3aede36 Oct 30 16:55:14.098277 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:14.098123 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5l2t5_be9a1b81-45d0-46a0-a771-84d88046f28e/kube-rbac-proxy-main/0.log" Oct 30 16:55:14.214924 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:14.214890 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5l2t5_be9a1b81-45d0-46a0-a771-84d88046f28e/kube-rbac-proxy-self/0.log" Oct 30 16:55:14.340597 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:14.340567 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5d79986586-bccmm_5beb53b1-b44b-47eb-84d8-31660627d4a3/metrics-server/0.log" Oct 30 16:55:14.589594 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:14.589568 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b9fbq_ae1acb11-b25d-43d5-b027-5edcf8e5fd1b/node-exporter/0.log" Oct 30 16:55:14.710592 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:14.710510 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b9fbq_ae1acb11-b25d-43d5-b027-5edcf8e5fd1b/kube-rbac-proxy/0.log" Oct 30 16:55:14.827364 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:14.827334 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b9fbq_ae1acb11-b25d-43d5-b027-5edcf8e5fd1b/init-textfile/0.log" Oct 30 16:55:15.019043 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:15.019001 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" event={"ID":"4a5872c4-c7aa-4c85-97fc-4b956ad824de","Type":"ContainerStarted","Data":"e5623a7c23aadc3a699d9ce790edc0e6ab632502aeef45f1ae206f155d5e9187"} Oct 30 16:55:15.019248 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:15.019068 5838 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" event={"ID":"4a5872c4-c7aa-4c85-97fc-4b956ad824de","Type":"ContainerStarted","Data":"af0161ca8e0a0b6c9d6af389b77db10b30aaefa9c294d10d8e8abf43b3aede36"} Oct 30 16:55:15.019248 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:15.019099 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:15.060375 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:15.060323 5838 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" podStartSLOduration=2.060309203 podStartE2EDuration="2.060309203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:15.058864608 +0000 UTC m=+2076.807453997" watchObservedRunningTime="2026-10-30 16:55:15.060309203 +0000 UTC m=+2076.808898587" Oct 30 16:55:15.761350 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:15.761319 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pdk47_3d0efab9-17c2-4f08-9433-02670766f69c/kube-rbac-proxy-main/0.log" Oct 30 16:55:15.881037 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:15.881011 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pdk47_3d0efab9-17c2-4f08-9433-02670766f69c/kube-rbac-proxy-self/0.log" Oct 30 16:55:16.004729 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:16.004698 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pdk47_3d0efab9-17c2-4f08-9433-02670766f69c/openshift-state-metrics/0.log" Oct 30 16:55:17.309544 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:17.309512 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f47789cbf-4q68x_51fcc480-c8f0-46f6-8092-04e12b809d95/telemeter-client/0.log" Oct 30 16:55:17.428873 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:17.428848 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f47789cbf-4q68x_51fcc480-c8f0-46f6-8092-04e12b809d95/reload/0.log" Oct 30 16:55:17.547582 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:17.547556 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f47789cbf-4q68x_51fcc480-c8f0-46f6-8092-04e12b809d95/kube-rbac-proxy/0.log" Oct 30 16:55:21.040668 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:21.040582 5838 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dcxv9/perf-node-gather-daemonset-l765s" Oct 30 16:55:26.361954 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:26.361912 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-vdpqj_5c2f91bc-56b8-45bb-9b39-876213305120/istio-proxy/0.log" Oct 30 16:55:26.420263 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:26.420228 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vp4m7_791cb17f-1c21-414c-9f90-196b26aa3570/dns/0.log" Oct 30 16:55:26.598728 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:26.598696 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vp4m7_791cb17f-1c21-414c-9f90-196b26aa3570/kube-rbac-proxy/0.log" Oct 30 16:55:26.777253 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:26.777182 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7rnq2_cd2b0c8e-8194-456d-8299-c43b1688404a/dns-node-resolver/0.log" Oct 30 16:55:27.495148 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:27.495103 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vl9xp_bb333f2b-2469-4dc5-bb87-2e633640b18b/discovery/0.log" Oct 30 16:55:27.527847 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:27.527814 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dcc5c66cd-7dvjr_035709ff-bb86-42f9-952b-95e17119d171/kube-auth-proxy/0.log" Oct 30 16:55:27.962623 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:27.962532 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-djmdh_3465e7a5-d6cc-4c8c-9052-29034ba0847f/istio-proxy/0.log" Oct 30 16:55:27.964872 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:27.964819 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-djmdh_3465e7a5-d6cc-4c8c-9052-29034ba0847f/istio-proxy/1.log" Oct 30 16:55:28.000846 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:28.000815 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-dxhn9_006df6fa-0ef9-4007-869e-0a251d92c372/istio-proxy/0.log" Oct 30 16:55:28.275344 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:28.275309 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f77ddbcb7-x6sns_fecda1cb-458a-4c3e-b48f-a0e6d82c1995/istio-proxy/0.log" Oct 30 16:55:28.402668 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:28.402626 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-sv5ln_8aeeae52-0ed0-4b70-9da0-640a6dfac794/payload-pre-processing/0.log" Oct 30 16:55:28.468315 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:28.468282 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-9hs5j_72f776e7-ae4d-4ccc-ab01-d107e80df07e/payload-processing/0.log" Oct 30 16:55:28.469929 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:28.469908 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-46t9k_fb2c72e6-372c-41c1-ba29-d47c0b521f34/node-ca/0.log" Oct 30 16:55:30.705179 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:30.705142 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-vdpqj_5c2f91bc-56b8-45bb-9b39-876213305120/istio-proxy/0.log" Oct 30 16:55:30.869017 ip-10-0-137-219 kubenswrapper[5838]: I1030 16:55:30.868959 5838 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67df64d57-vdpqj_5c2f91bc-56b8-45bb-9b39-876213305120/istio-proxy/0.log" Oct 30 16:55:31.808013 ip-10-0-137-219 kubenswrapper[5838]: E1030 16:55:31.807959 5838 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n8h27" containerName="extractor"